Software Diagnostics Library

Crash Dump Analysis Patterns (Part 240)

Sunday, May 29th, 2016

Windows processes may contain Execution Residue such as ASCII window class names in mapped memory regions pointing to other running processes (perhaps as a result of Hooksware). For example, calc.exe process memory dump saved on my Windows 10 notebook “knows” about Visio and WinDbg windows that were opened at that time:

0:000> s-a 0 L?FFFFFFFFFFFFFFFF "VISIOA" 00000015`42c6bdd0 56 49 53 49 4f 41 00 00-00 00 00 00 00 00 00 00 VISIOA.............

0:000> s-a 0 L?FFFFFFFFFFFFFFFF "WinDbg" 00000015`42d19720 57 69 6e 44 62 67 46 72-61 6d 65 43 6c 61 73 73 WinDbgFrameClass

This may be useful for some troubleshooting scenarios, for example, pointing to processes which are known for their problematic behavior or Special Processes. Of course, we assume that those windows or classes were genuine, not faked. We call this analysis pattern Window Hint similar to Environment Hint and Module Hint analysis patterns.

Going deeper, we can dump strings from the whole region limiting the output to the strings with length more than 5:

0:000> !address 00000015`42d19720

Usage: <unknown> Base Address: 00000015`42b20000 End Address: 00000015`42d3a000 Region Size: 00000000`0021a000 ( 2.102 MB) State: 00001000 MEM_COMMIT Protect: 00000002 PAGE_READONLY Type: 00040000 MEM_MAPPED Allocation Base: 00000015`42b20000 Allocation Protect: 00000002 PAGE_READONLY

Content source: 1 (target), length: 208e0

0:000> s-[l5]sa 00000015`42b20000 00000015`42d3a000 00000015`42b20a60 “#32769″ 00000015`42b20cc0 “Message” 00000015`42b20f40 “#32774″ 00000015`42b21060 “#32772″ 00000015`42b21510 “Ghost” 00000015`42b215e0 “LivePreview” 00000015`42b216f0 “UserAdapterWindowClass” 00000015`42b21ce0 “MSCTFIME Composition” 00000015`42b222a0 “#32772″ 00000015`42b22390 “#32772″ 00000015`42b22460 “RichEdit20W” 00000015`42b22530 “RichEdit20A” 00000015`42b22600 “ToolbarWindow32″ 00000015`42b226e0 “tooltips_class32″ 00000015`42b227c0 “msctls_statusbar32″ 00000015`42b228a0 “SysListView32″ 00000015`42b22980 “SysHeader32″ 00000015`42b22a50 “SysTabControl32″ 00000015`42b22b30 “SysTreeView32″ 00000015`42b22c10 “msctls_trackbar32″ 00000015`42b22cf0 “msctls_updown32″ 00000015`42b22dd0 “msctls_progress32″ 00000015`42b22eb0 “msctls_hotkey32″ 00000015`42b22f8f “‘SysAnimate32″ 00000015`42b230f0 “SysIPAddress32″ 00000015`42b231d0 “ReBarWindow32″ 00000015`42b232b0 “ComboBoxEx32″ 00000015`42b23390 “SysMonthCal32″ 00000015`42b23470 “SysDateTimePick32″ 00000015`42b23550 “DropDown” 00000015`42b23620 “SysLink” 00000015`42b236f0 “SysPager” 00000015`42b23960 “msctls_netaddress”

[...]

00000015`42d175e0 "OutlookFbThreadWnd" 00000015`42d19720 "WinDbgFrameClass" 00000015`42d19750 "DockClass" 00000015`42d19770 "GhostClass" 00000015`42d19a30 "ATL:00007FF60D792730" 00000015`42d1a0f0 "MSCTFIME Composition" 00000015`42d1a4af "%OleMainThreadWndClass" 00000015`42d1be10 "CicMarshalWndClass" 00000015`42d1c0e0 "VSyncHelper-00000040EC4CA5F0-1f8" 00000015`42d1c100 "8855daf" 00000015`42d1c190 "URL Moniker Notification Window" 00000015`42d1c390 "UserAdapterWindowClass" 00000015`42d1d080 "@>zG#" 00000015`42d1dcaf "!VSyncHelper-00000040D60C5850-1e" 00000015`42d1dccf "ef0477df" 00000015`42d20d50 "VSyncHelper-00000040F39C5650-1f0" 00000015`42d20d70 "313c5a0" 00000015`42d250d0 "#32770" 00000015`42d250f0 "URL Moniker Notification Window" 00000015`42d29270 "VSyncHelper-00000079321C32E0-1f2" 00000015`42d29290 "fb11f8c" 00000015`42d2a1d0 "MSCTFIME Composition" 00000015`42d2a480 "CicMarshalWndClass" 00000015`42d2ac80 "MSCTFIME Composition" 00000015`42d2b8d0 "ShockwaveFlashFullScreen" 00000015`42d2bbb8 "P?U!\" 00000015`42d2c690 "Xaml_WindowedPopupClass" 00000015`42d30a10 "ShockwaveFlashFullScreen" 00000015`42d30b50 "MSCTFIME UI" 00000015`42d30b90 "WinBaseClass" 00000015`42d3441f "!Alternate Owner" 00000015`42d34460 "ShockwaveFlashFullScreen" 00000015`42d344a0 "ATL:00007FF60D792530" 00000015`42d34a50 "SysAnimate32" 00000015`42d34a7f "'ComboBoxEx32" 00000015`42d34ed0 "tooltips_class32" 00000015`42d34f00 "msctls_statusbar32" 00000015`42d35e70 "RawInputClass" 00000015`42d36a10 "SysTabControl32" 00000015`42d38650 "CicMarshalWndClass" 00000015`42d38eb0 "#32772" 00000015`42d3951f "!VSyncHelper-000000C9DA06CD10-1f" 00000015`42d3953f "110e8d16"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Memory Forensics, Windows 10, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 239)

Monday, May 2nd, 2016

We found out that in Windows 10 (at least on our working system) Notepad is no longer a single threaded application even without opening any common dialogs (like in Evental Dumps analysis pattern example). It has at least 3 additional threads (and other modeling applications we use for our training also have additional threads):

0:000> ~*k

0 Id: 3a64.3b38 Suspend: 1 Teb: 00007ff6`a914d000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`6298f938 00007ffa`e57cf8e5 USER32!NtUserGetMessage+0xa 01 000000e5`6298f940 00007ff6`a9603470 USER32!GetMessageW+0x25 02 000000e5`6298f970 00007ff6`a96141f5 notepad!WinMain+0x178 03 000000e5`6298f9f0 00007ffa`e3b42d92 notepad!WinMainCRTStartup+0x1c5 04 000000e5`6298fab0 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 05 000000e5`6298fae0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

1 Id: 3a64.38b0 Suspend: 1 Teb: 00007ff6`a914b000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`62bffa58 00007ffa`e5bf93a5 ntdll!NtWaitForWorkViaWorkerFactory+0xa 01 000000e5`62bffa60 00007ffa`e3b42d92 ntdll!TppWorkerThread+0x295 02 000000e5`62bffe60 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 03 000000e5`62bffe90 00000000`00000000 ntdll!RtlUserThreadStart+0x34

2 Id: 3a64.3940 Suspend: 1 Teb: 00007ff6`a9149000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`62c7f718 00007ffa`e5bf93a5 ntdll!NtWaitForWorkViaWorkerFactory+0xa 01 000000e5`62c7f720 00007ffa`e3b42d92 ntdll!TppWorkerThread+0x295 02 000000e5`62c7fb20 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 03 000000e5`62c7fb50 00000000`00000000 ntdll!RtlUserThreadStart+0x34

3 Id: 3a64.1030 Suspend: 1 Teb: 00007ff6`a9147000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`62d1f878 00007ffa`e5bf93a5 ntdll!NtWaitForWorkViaWorkerFactory+0xa 01 000000e5`62d1f880 00007ffa`e3b42d92 ntdll!TppWorkerThread+0x295 02 000000e5`62d1fc80 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 03 000000e5`62d1fcb0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

This gave us an idea for the analysis pattern we call Not My Thread since additional threads can be started by any other process DLLs, for example, by Hooksware. However, we need to distinguish between unexpectedly added threads, threads with Special Stack Traces and Special Threads, for example, from .NET support.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Windows 10, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 238)

Thursday, March 17th, 2016

Sometimes developers introduce their own variants of synchronization code instead of using synchronization API provided by language runtime and OS. If we are lucky we can spot it in function and class method names and then use Constant Subtrace analysis pattern:

0: kd> kc *** Stack trace for last set context - .thread/.cxr resets it # Call Site 00 nt!KiSwapContext 01 nt!KiCommitThreadWait 02 nt!KeWaitForSingleObject 03 nt!NtWaitForSingleObject 04 nt!KiSystemServiceCopyEnd 05 ntdll!ZwWaitForSingleObject 06 KERNELBASE!WaitForSingleObjectEx 07 wbemcomn!CWbemCriticalSection::Enter 08 wbemcore!EnsureInitialized 09 wbemcore!InitAndWaitForClient 0a wbemcore!CWbemLevel1Login::ConnectorLogin 0b wbemcore!CWbemLevel1Login::NTLMLogin 0c RPCRT4!Invoke 0d RPCRT4!NdrStubCall2 0e ole32!CStdStubBuffer_Invoke 0f ole32!SyncStubInvoke 10 ole32!StubInvoke 11 ole32!CCtxComChnl::ContextInvoke 12 ole32!AppInvoke 13 ole32!ComInvokeWithLockAndIPID 14 ole32!ThreadInvoke 15 RPCRT4!DispatchToStubInCNoAvrf 16 RPCRT4!RPC_INTERFACE::DispatchToStubWorker 17 RPCRT4!RPC_INTERFACE::DispatchToStub 18 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject 19 RPCRT4!LRPC_SCALL::DispatchRequest 1a RPCRT4!LRPC_SCALL::HandleRequest 1b RPCRT4!LRPC_SASSOCIATION::HandleRequest 1c RPCRT4!LRPC_ADDRESS::HandleRequest 1d RPCRT4!LRPC_ADDRESS::ProcessIO 1e RPCRT4!LrpcIoComplete 1f ntdll!TppAlpcpExecuteCallback 20 ntdll!TppWorkerThread 21 kernel32!BaseThreadInitThunk 22 ntdll!RtlUserThreadStart

0: kd> kc *** Stack trace for last set context - .thread/.cxr resets it # Call Site 00 repdrvfs!SCachePage::operator= 01 repdrvfs!std::vector<scachepage,wbem_allocator<scachepage> >::erase 02 repdrvfs!CPageCache::Read 03 repdrvfs!CPageFile::GetPage 04 repdrvfs!ValidateBTreeAgainstObjHeap 05 repdrvfs!PerformAllValidations 06 repdrvfs!VerifyRepositoryOnline 07 repdrvfs!VerifyRepository 08 repdrvfs!CPageSource::Startup 09 repdrvfs!CPageSource::Init 0a repdrvfs!CFileCache::InnerInitialize 0b repdrvfs!CFileCache::Initialize 0c repdrvfs!CRepository::Initialize 0d repdrvfs!CRepository::Logon 0e wbemcore!CRepository::Init 0f wbemcore!InitSubsystems 10 wbemcore!ConfigMgr::InitSystem 11 wbemcore!EnsureInitialized 12 wbemcore!InitAndWaitForClient 13 wbemcore!CWbemLevel1Login::ConnectorLogin 14 wbemcore!CWbemLevel1Login::NTLMLogin 15 RPCRT4!Invoke 16 RPCRT4!NdrStubCall2 17 ole32!CStdStubBuffer_Invoke 18 ole32!SyncStubInvoke 19 ole32!StubInvoke 1a ole32!CCtxComChnl::ContextInvoke 1b ole32!AppInvoke 1c ole32!ComInvokeWithLockAndIPID 1d ole32!ThreadInvoke 1e RPCRT4!DispatchToStubInCNoAvrf 1f RPCRT4!RPC_INTERFACE::DispatchToStubWorker 20 RPCRT4!RPC_INTERFACE::DispatchToStub 21 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject 22 RPCRT4!LRPC_SCALL::DispatchRequest 23 RPCRT4!LRPC_SCALL::HandleRequest 24 RPCRT4!LRPC_SASSOCIATION::HandleRequest 25 RPCRT4!LRPC_ADDRESS::HandleRequest 26 RPCRT4!LRPC_ADDRESS::ProcessIO 27 RPCRT4!LrpcIoComplete 28 ntdll!TppAlpcpExecuteCallback 29 ntdll!TppWorkerThread 2a kernel32!BaseThreadInitThunk 2b ntdll!RtlUserThreadStart

These two thread stack traces were spotted from a complete memory dump Stack Trace Collection as the part of a larger ALPC Wait Chain. We switched to these threads using .thread /r /p WinDbg command to get the stripped stack trace via kc command for better illustration. We see Constant Subtrace until wbemcore!EnsureInitialized function which serves as a bifurcation stack frame. The first stack trace has “CriticalSection::Enter” after the bifurcation stack frame compared to the second stack trace which looks like Spiking Thread in user space.

There is no hidden critical section associated with that process except the one which is probably related to the spiking Variable Subtrace since it doesn’t have any LockCount:

0: kd> !cs -l -o -s DebugInfo = 0x0000000004a774d0 Critical section = 0x000000000308d690 (+0x308D690) LOCKED LockCount = 0×0 WaiterWoken = No OwningThread = 0×0000000000000928 RecursionCount = 0×1 LockSemaphore = 0×0 SpinCount = 0×0000000000000000 OwningThread = .thread fffffa806ebd8060 ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled

We can also disassemble wbemcomn!CWbemCriticalSection::Enter and find out that it calls WaitForSingleObject once and no other synchronization API indeed:

0: kd> uf wbemcomn!CWbemCriticalSection::Enter [...] wbemcomn!CWbemCriticalSection::Enter+0x41: 000007fe`f78ad1c0 mov rcx,qword ptr [rbx+10h] 000007fe`f78ad1c4 mov edx,r12d 000007fe`f78ad1c7 call qword ptr [wbemcomn!_imp_WaitForSingleObject (000007fe`f78ee4f0)] 000007fe`f78ad1cd cmp eax,esi 000007fe`f78ad1cf je wbemcomn!CWbemCriticalSection::Enter+0x52 (000007fe`f78a62a3) Branch

We add this nonstandard synchronization memory analysis pattern to Wait Chain analysis pattern category.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 233)

Wednesday, November 18th, 2015

With the possibility of process cloning (reflection) starting from Windows 7 it is possible to get memory snapshots (Clone Dump) from a process clone (similar to fork API in Unix). Procdump tool has -r switch for that purpose. We checked this with x64 Windows 7 notepad.exe. We got two memory dumps: one is a clone with this stack trace:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dbgcfg.dmp] User Mini Dump File with Full Memory: Only application data is available

Comment: ' *** procdump -ma -r Notepad.exe *** Manual dump'

0:000> ~*k

. 0 Id: 25ec.147c Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Child-SP RetAddr Call Site 00 00000000`02c8fd38 00000000`7733aae7 ntdll!NtSuspendThread+0xa 01 00000000`02c8fd40 00000000`77165a4d ntdll!RtlpProcessReflectionStartup+0×2e7 02 00000000`02c8fe30 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd 03 00000000`02c8fe60 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The process memory has all address space of the original process including module list and heap structure:

0:000> lmn start end module name 00000000`77050000 00000000`7714a000 user32 user32.dll 00000000`77150000 00000000`77270000 kernel32 kernel32.dll 00000000`77270000 00000000`77419000 ntdll ntdll.dll 00000000`ff030000 00000000`ff065000 notepad notepad.exe 000007fe`f57d0000 000007fe`f5841000 winspool winspool.drv 000007fe`fb730000 000007fe`fb786000 uxtheme uxtheme.dll 000007fe`fb910000 000007fe`fbb04000 comctl32 comctl32.dll 000007fe`fbf00000 000007fe`fbf0c000 version version.dll 000007fe`fceb0000 000007fe`fcebf000 CRYPTBASE CRYPTBASE.dll 000007fe`fd310000 000007fe`fd37c000 KERNELBASE KERNELBASE.dll 000007fe`fd3d0000 000007fe`fd499000 usp10 usp10.dll 000007fe`fd4a0000 000007fe`fd511000 shlwapi shlwapi.dll 000007fe`fd520000 000007fe`fd64d000 rpcrt4 rpcrt4.dll 000007fe`fd650000 000007fe`fd66f000 sechost sechost.dll 000007fe`fd680000 000007fe`fd6ae000 imm32 imm32.dll 000007fe`fd6b0000 000007fe`fd78b000 advapi32 advapi32.dll 000007fe`fd810000 000007fe`fd8a7000 comdlg32 comdlg32.dll 000007fe`fdd60000 000007fe`fddff000 msvcrt msvcrt.dll 000007fe`fdfe0000 000007fe`fed69000 shell32 shell32.dll 000007fe`fed70000 000007fe`fedd7000 gdi32 gdi32.dll 000007fe`ff0b0000 000007fe`ff1b9000 msctf msctf.dll 000007fe`ff1c0000 000007fe`ff1ce000 lpk lpk.dll 000007fe`ff1d0000 000007fe`ff2a7000 oleaut32 oleaut32.dll 000007fe`ff2b0000 000007fe`ff349000 clbcatq clbcatq.dll 000007fe`ff350000 000007fe`ff553000 ole32 ole32.dll

0:000> !address -summary

Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 48 7ff`faa06000 ( 8.000 TB) 100.00% Image 129 0`01e79000 ( 30.473 MB) 35.47% 0.00% 21 0`01d10000 ( 29.063 MB) 33.83% 0.00% Other 9 0`016be000 ( 22.742 MB) 26.47% 0.00% Heap 26 0`00320000 ( 3.125 MB) 3.64% 0.00% Stack 3 0`00080000 ( 512.000 kB) 0.58% 0.00% TEB 1 0`00002000 ( 8.000 kB) 0.01% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_IMAGE 130 0`01e7a000 ( 30.477 MB) 35.47% 0.00% MEM_PRIVATE 47 0`01449000 ( 20.285 MB) 23.61% 0.00% MEM_MAPPED 11 0`00c97000 ( 12.590 MB) 14.65% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 50 7ff`fc096000 ( 8.000 TB) 100.00% MEM_COMMIT 176 0`02c10000 ( 44.063 MB) 51.29% 0.00% MEM_RESERVE 12 0`0134a000 ( 19.289 MB) 22.45% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READONLY 83 0`01b46000 ( 27.273 MB) 31.75% 0.00% PAGE_EXECUTE_READ 25 0`00f6d000 ( 15.426 MB) 17.95% 0.00% PAGE_WRITECOPY 48 0`00126000 ( 1.148 MB) 1.34% 0.00% PAGE_READWRITE 18 0`00032000 ( 200.000 kB) 0.23% 0.00% PAGE_READWRITE|PAGE_GUARD 2 0`00005000 ( 20.000 kB) 0.02% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 0`ff065000 7fd`f676b000 ( 7.992 TB) Image 7fe`fe4cb000 0`0089e000 ( 8.617 MB) 0`7f0e0000 0`00f00000 ( 15.000 MB) Other 0`00610000 0`01590000 ( 21.563 MB) Heap 0`003b8000 0`000c8000 ( 800.000 kB) Stack 0`02c10000 0`0006c000 ( 432.000 kB) TEB 7ff`fffdb000 0`00002000 ( 8.000 kB) PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

0:000> !heap -s

****************************************************** NT HEAP STATS BELOW ****************************************************** LFH Key : 0x000000381021167d Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ------------------------------------------------------------------------------------- 0000000000280000 00000002 1024 412 1024 14 4 1 0 0 LFH 0000000000250000 00001002 1088 256 1088 5 2 2 0 0 LFH 0000000001cc0000 00001002 64 8 64 3 1 1 0 0 0000000001e60000 00001002 512 120 512 49 3 1 0 0 0000000001dc0000 00001002 512 8 512 2 1 1 0 0 -------------------------------------------------------------------------------------

The other dump saved is a minidump from which we can get thread information for Execution Residue (raw stack data) and reconstruct stack traces in Clone Dump:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dmp] Comment: ' *** procdump -ma -r Notepad.exe *** Manual dump' User Mini Dump File: Only registers, stack and portions of memory are available

0:000> ~ . 0 Id: 87c.27f4 Suspend: 0 Teb: 000007ff`fffdd000 Unfrozen

0:000> k # Child-SP RetAddr Call Site 00 00000000`0016fac8 00000000`77069e9e 0x77069e6a 01 00000000`0016fad0 00000000`00000000 0x77069e9e

0:000> r rax=0000000000000000 rbx=000000000016fb40 rcx=0000000000280000 rdx=0000000000000000 rsi=0000000000000001 rdi=0000000000000000 rip=0000000077069e6a rsp=000000000016fac8 rbp=00000000ff030000 r8=000000000016f8e8 r9=00000000000a0cdc r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 00000000`77069e6a c3 ret

Now we can see the original stack trace in Clone Dump:

0:000> k =000000000016fac8 ff # Child-SP RetAddr Call Site 00 00000000`0016fac8 00000000`77069e9e ntdll!NtSuspendThread+0xa 01 00000000`0016fad0 00000000`ff031064 user32!GetMessageW+0x34 02 00000000`0016fb00 00000000`ff03133c notepad!WinMain+0x182 03 00000000`0016fb80 00000000`77165a4d notepad!DisplayNonGenuineDlgWorker+0x2da 04 00000000`0016fc40 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd 05 00000000`0016fc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Since we know TEB address from the minidump we can get stack region boundaries in Clone Dump:

0:000> dt _NT_TIB 000007fffffdd000 ntdll!_NT_TIB +0x000 ExceptionList : (null) +0x008 StackBase : 0x00000000`00170000 Void +0x010 StackLimit : 0x00000000`0015b000 Void +0x018 SubSystemTib : (null) +0x020 FiberData : 0x00000000`00001e00 Void +0x020 Version : 0x1e00 +0x028 ArbitraryUserPointer : (null) +0x030 Self : 0x000007ff`fffdd000 _NT_TIB

Now we can check Execution Residue (for example, for signs of Hidden Exceptions):

0:000> dpS 0x00000000`0015b000 0x00000000`00170000 [...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 232)

Saturday, October 31st, 2015

We already introduced Active Thread pattern variant for Mac OS X. Here we provide an example for Windows. Unless we have Evental Dumps, Active Threads in Windows are usually threads from Busy System or Spiking Threads, and, therefore, represent an abnormal behavior since most threads are waiting or calling some API. For Evental Dumps they may be just normal threads:

0:000> r rax=0000000000000006 rbx=0000000000000003 rcx=0000000000000018 rdx=0000000000000000 rsi=000000000028c601 rdi=0000000002bee25e rip=000007feff1a5a09 rsp=000000000028c380 rbp=0000000000000000 r8=0000000000000000 r9=00000000001653a0 r10=000000000000000e r11=000000000000000a r12=0000000000000006 r13=000000000028ca38 r14=0000000002bec888 r15=0000000000173630 iopl=0 nv up ei pl nz ac po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216 usp10!otlChainingLookup::apply+0×2f9: 000007fe`ff1a5a09 498d0c06 lea rcx,[r14+rax]

0:000> k # Child-SP RetAddr Call Site 00 00000000`0028c380 000007fe`ff19f4f2 usp10!otlChainingLookup::apply+0×2f9 01 00000000`0028c4b0 000007fe`ff19e777 usp10!ApplyLookup+0×592 02 00000000`0028c5a0 000007fe`ff19a634 usp10!ApplyFeatures+0×777 03 00000000`0028c860 000007fe`ff181800 usp10!SubstituteOtlGlyphs+0×224 04 00000000`0028c910 000007fe`ff174cc0 usp10!GenericEngineGetGlyphs+0×1000 05 00000000`0028ccb0 000007fe`ff1389c5 usp10!ShlShape+0×7a0 06 00000000`0028ced0 000007fe`ff147363 usp10!ScriptShape+0×205 07 00000000`0028cf70 000007fe`ff148ac9 usp10!RenderItemNoFallback+0×433 08 00000000`0028d030 000007fe`ff148d86 usp10!RenderItemWithFallback+0×129 09 00000000`0028d080 000007fe`ff14a5f7 usp10!RenderItem+0×36 0a 00000000`0028d0d0 000007fe`ff13b2c9 usp10!ScriptStringAnalyzeGlyphs+0×277 0b 00000000`0028d170 000007fe`fdd616bf usp10!ScriptStringAnalyse+0×399 0c 00000000`0028d1f0 000007fe`fdd614cc lpk!LpkCharsetDraw+0×4eb 0d 00000000`0028d3b0 00000000`774e85c5 lpk!LpkDrawTextEx+0×68 0e 00000000`0028d420 00000000`774e865c user32!DT_DrawStr+0xa6 0f 00000000`0028d4c0 00000000`774e826c user32!DT_DrawJustifiedLine+0xa6 10 00000000`0028d530 00000000`774e6cc8 user32!DrawTextExWorker+0×442 11 00000000`0028d640 000007fe`fbd840d1 user32!DrawTextW+0×57 12 00000000`0028d6b0 000007fe`fbd83e49 comctl32!CLVView::_ComputeLabelSizeWorker+0×1d1 13 00000000`0028da40 000007fe`fbd8cc48 comctl32!CLVView::v_RecomputeLabelSize+0×1f9 14 00000000`0028dd70 000007fe`fbda9d24 comctl32!CLVListView::v_DrawItem+0×284 15 00000000`0028e110 000007fe`fbd9773b comctl32!CLVDrawItemManager::DrawItem+0×4c0 16 00000000`0028e170 000007fe`fbd95f8e comctl32!CLVDrawManager::_PaintItems+0×3df 17 00000000`0028e3b0 000007fe`fbd95e87 comctl32!CLVDrawManager::_PaintWorkArea+0xda 18 00000000`0028e430 000007fe`fbd95cff comctl32!CLVDrawManager::_OnPaintWorkAreas+0×147 19 00000000`0028e4c0 000007fe`fbd06f1b comctl32!CLVDrawManager::_OnPaint+0×14b 1a 00000000`0028e570 000007fe`fbd06011 comctl32!CListView::WndProc+0xebf 1b 00000000`0028e770 00000000`774e9bd1 comctl32!CListView::s_WndProc+0×6cd 1c 00000000`0028e7d0 00000000`774e3bfc user32!UserCallWinProcCheckWow+0×1ad 1d 00000000`0028e890 00000000`774e3b78 user32!CallWindowProcAorW+0xdc 1e 00000000`0028e8e0 000007fe`fbca6215 user32!CallWindowProcW+0×18 1f 00000000`0028e920 000007fe`fbca69a0 comctl32!CallOriginalWndProc+0×1d 20 00000000`0028e960 000007fe`fbca6768 comctl32!CallNextSubclassProc+0×8c 21 00000000`0028e9e0 000007fe`fde1096a comctl32!DefSubclassProc+0×7c 22 00000000`0028ea30 000007fe`fdde9df4 shell32!DefSubclassProc+0×56 23 00000000`0028ea60 000007fe`fbca69a0 shell32!CListViewHost::s_ListViewSubclassWndProc+0×267 24 00000000`0028eb40 000007fe`fbca6877 comctl32!CallNextSubclassProc+0×8c 25 00000000`0028ebc0 00000000`774e9bd1 comctl32!MasterSubclassProc+0xe7 26 00000000`0028ec60 00000000`774e72cb user32!UserCallWinProcCheckWow+0×1ad 27 00000000`0028ed20 00000000`774e6829 user32!DispatchClientMessage+0xc3 28 00000000`0028ed80 00000000`776211f5 user32!_fnDWORD+0×2d 29 00000000`0028ede0 00000000`774e6e5a ntdll!KiUserCallbackDispatcherContinue 2a 00000000`0028ee68 00000000`774e6e6c user32!NtUserDispatchMessage+0xa 2b 00000000`0028ee70 00000000`774e67c2 user32!DispatchMessageWorker+0×55b 2c 00000000`0028eef0 000007fe`fbcc34a4 user32!IsDialogMessageW+0×153 2d 00000000`0028ef80 000007fe`fbcc583f comctl32!Prop_IsDialogMessage+0×1f0 2e 00000000`0028eff0 000007fe`fbcc5c05 comctl32!_RealPropertySheet+0×31b 2f 00000000`0028f0c0 000007fe`ff214e68 comctl32!_PropertySheet+0×55 30 00000000`0028f100 000007fe`ff214bb1 comdlg32!Print_InvokePropertySheets+0×2c6 31 00000000`0028f660 000007fe`ff21499c comdlg32!PrintDlgExX+0×2be 32 00000000`0028f6c0 00000000`ffec250f comdlg32!PrintDlgExW+0×38 33 00000000`0028f730 00000000`ffec242b notepad!GetPrinterDCviaDialog+0xab 34 00000000`0028f7f0 00000000`ffec23e8 notepad!PrintIt+0×46 35 00000000`0028fb70 00000000`ffec14eb notepad!NPCommand+0xdb 36 00000000`0028fca0 00000000`774e9bd1 notepad!NPWndProc+0×540 37 00000000`0028fce0 00000000`774e98da user32!UserCallWinProcCheckWow+0×1ad 38 00000000`0028fda0 00000000`ffec10bc user32!DispatchMessageWorker+0×3b5 39 00000000`0028fe20 00000000`ffec133c notepad!WinMain+0×16f 3a 00000000`0028fea0 00000000`773c59ed notepad!DisplayNonGenuineDlgWorker+0×2da 3b 00000000`0028ff60 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 3c 00000000`0028ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see the thread is active, in the middle of the function. For comparison, the next 2 threads are waiting and calling API respectively:

0:000> ~1k # Child-SP RetAddr Call Site 00 00000000`02edf748 00000000`775eb037 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`02edf750 00000000`773c59ed ntdll!TppWaiterpThread+0×14d 02 00000000`02edf9f0 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 03 00000000`02edfa20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ~2k # Child-SP RetAddr Call Site 00 00000000`0344e048 000007fe`fd3f403e ntdll!NtUnmapViewOfSection+0xa 01 00000000`0344e050 00000000`774e2edf KERNELBASE!FreeLibrary+0xa4 02 00000000`0344e080 000007fe`fdddaab3 user32!PrivateExtractIconsW+0×34b 03 00000000`0344e5a0 000007fe`fdddac28 shell32!SHPrivateExtractIcons+0×50a 04 00000000`0344e870 000007fe`fdde34b4 shell32!SHDefExtractIconW+0×254 05 00000000`0344eb60 000007fe`fdde3435 shell32!CExtractIcon::_ExtractW+0xcd 06 00000000`0344ebe0 000007fe`fdf0d529 shell32!CExtractIconBase::Extract+0×21 07 00000000`0344ec20 000007fe`fdf0d2da shell32!IExtractIcon_Extract+0×43 08 00000000`0344ec60 000007fe`fdddfff0 shell32!_GetILIndexGivenPXIcon+0×22e 09 00000000`0344f100 000007fe`fdde27a4 shell32!_GetILIndexFromItem+0×87 0a 00000000`0344f1a0 000007fe`fddb6506 shell32!SHGetIconIndexFromPIDL+0×66 0b 00000000`0344f1d0 000007fe`fdedb186 shell32!MapIDListToIconILIndex+0×52 0c 00000000`0344f250 000007fe`fdddc54c shell32!CLoadSystemIconTask::InternalResumeRT+0×110 0d 00000000`0344f2e0 000007fe`fde0efcb shell32!CRunnableTask::Run+0xda 0e 00000000`0344f310 000007fe`fde12b56 shell32!CShellTask::TT_Run+0×124 0f 00000000`0344f340 000007fe`fde12cb2 shell32!CShellTaskThread::ThreadProc+0×1d2 10 00000000`0344f3e0 000007fe`ff2b3843 shell32!CShellTaskThread::s_ThreadProc+0×22 11 00000000`0344f410 00000000`775f15db shlwapi!ExecuteWorkItemThreadProc+0xf 12 00000000`0344f440 00000000`775f0c56 ntdll!RtlpTpWorkCallback+0×16b 13 00000000`0344f520 00000000`773c59ed ntdll!TppWorkerThread+0×5ff 14 00000000`0344f820 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 15 00000000`0344f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub ntdll!NtUnmapViewOfSection+0xa ntdll!NtAccessCheckAndAuditAlarm: 00000000`77621540 4c8bd1 mov r10,rcx 00000000`77621543 b826000000 mov eax,26h 00000000`77621548 0f05 syscall 00000000`7762154a c3 ret 00000000`7762154b 0f1f440000 nop dword ptr [rax+rax] ntdll!NtUnmapViewOfSection: 00000000`77621550 4c8bd1 mov r10,rcx 00000000`77621553 b827000000 mov eax,27h 00000000`77621558 0f05 syscall

Our Active Thread is not Spiking Thread since CPU consumption is minimal:

0:000> !runaway f User Mode Time Thread Time 0:1ca4 0 days 0:00:00.171 11:1fb0 0 days 0:00:00.000 10:f98 0 days 0:00:00.000 9:eb8 0 days 0:00:00.000 8:1b80 0 days 0:00:00.000 7:139c 0 days 0:00:00.000 6:1d9c 0 days 0:00:00.000 5:1b44 0 days 0:00:00.000 4:1edc 0 days 0:00:00.000 3:830 0 days 0:00:00.000 2:1638 0 days 0:00:00.000 1:1ab0 0 days 0:00:00.000 Kernel Mode Time Thread Time 0:1ca4 0 days 0:00:00.421 11:1fb0 0 days 0:00:00.000 10:f98 0 days 0:00:00.000 9:eb8 0 days 0:00:00.000 8:1b80 0 days 0:00:00.000 7:139c 0 days 0:00:00.000 6:1d9c 0 days 0:00:00.000 5:1b44 0 days 0:00:00.000 4:1edc 0 days 0:00:00.000 3:830 0 days 0:00:00.000 2:1638 0 days 0:00:00.000 1:1ab0 0 days 0:00:00.000 Elapsed Time Thread Time 11:1fb0 24692 days 13:29:46.335 0:1ca4 0 days 0:02:39.671 1:1ab0 0 days 0:01:48.239 2:1638 0 days 0:01:18.837 4:1edc 0 days 0:01:18.697 3:830 0 days 0:01:18.697 5:1b44 0 days 0:01:18.497 6:1d9c 0 days 0:01:18.387 7:139c 0 days 0:01:15.957 8:1b80 0 days 0:01:14.397 9:eb8 0 days 0:01:01.485 10:f98 0 days 0:00:39.849

However, the huge Elapsed Time for the thread #11 (most likely the value is uninitialized) and its stack trace suggest that the dump was saved on Create Thread debugging event by a debugger:

0:000> ~11k # Child-SP RetAddr Call Site 00 00000000`0666fb08 00000000`00000000 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 231)

Monday, October 26th, 2015

One of the customers of Software Diagnostics Services submitted memory dumps saved by DebugDiag to accompany software logs for the analysis of sudden process exit. We didn’t request such memory dumps and initially dismissed them. However, during software log analysis we decided to look at Adjoint Spaces to see whether there was some additional information in stack traces. We found out that those dumps were saved on each thread exit event. Since other threads were either waiting or Active Threads their analysis gave clues of process behavior before process exit. For example, we found ALPC Wait Chain to Coupled Process. The latter prompted us to analyze Coupled Activities in the software log and diagnose the possible problem there. Since saving memory dumps on thread creation and exit can be useful technique we decided to add Evental Dumps memory analysis pattern to our pattern catalog.

To illustrate this pattern we show Stack Trace Collection from notepad.exe. This process usually has just one thread. But, if we try to open a Print dialog the number of threads increases up to 12.

We attach WinDbg to notepad.exe process and set up debugging event filter (Debug \ Event Filters… menu) for Create Thread event with a command line as shown in this picture:

We then resume execution using g command and switch to Notepad. There we first open File \ Page Setup… dialog. We observe that a memory dump is saved. Then we open File \ Print… dialog and notice the creation 11 more process memory dump:

We now show stack traces from these dumps where we use ~*kc WinDbg command to minimize the amount of inessential for our purposes output:

// 1st dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!NtAlertThread 01 ntdll!TppWaiterEnqueueTransition 02 ntdll!TppWaitpSet 03 ntdll!TpSetWait 04 ntdll!TppTimerpInitTimerQueueQueue 05 ntdll!TppTimerpAllocTimerQueue 06 ntdll!TppTimerpAcquirePoolTimerQueue 07 ntdll!TppTimerAlloc 08 ntdll!TpAllocTimer 09 KERNELBASE!CreateThreadpoolTimer 0a rpcrt4!RPC_THREAD_POOL::CreateTimer 0b rpcrt4!GarbageCollectionNeeded 0c rpcrt4!LRPC_CASSOCIATION::RemoveReference 0d rpcrt4!LRPC_CCALL::`vector deleting destructor' 0e rpcrt4!LRPC_CCALL::FreeBuffer 0f rpcrt4!Ndr64pClientFinally 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 sechost!LsaLookupClose 13 sechost!LookupAccountNameInternal 14 sechost!LookupAccountNameLocalW 15 rpcrt4!RpcpLookupAccountNameDirect 16 rpcrt4!RpcpLookupAccountName 17 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 18 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 19 rpcrt4!RpcBindingSetAuthInfoExW 1a winspool!STRING_HANDLE_bind 1b rpcrt4!GenericHandleMgr 1c rpcrt4!ExplicitBindHandleMgr 1d rpcrt4!Ndr64pClientSetupTransferSyntax 1e rpcrt4!NdrpClientCall3 1f rpcrt4!NdrClientCall3 20 winspool!RpcSplOpenPrinter 21 winspool!OpenPrinterRPC 22 winspool!OpenPrinter2W 23 comdlg32!PrintOpenPrinter 24 comdlg32!PrintDlgX 25 comdlg32!PageSetupDlgX 26 comdlg32!PageSetupDlgW 27 notepad!NPCommand 28 notepad!NPWndProc 29 user32!UserCallWinProcCheckWow 2a user32!DispatchMessageWorker 2b notepad!WinMain 2c notepad!DisplayNonGenuineDlgWorker 2d kernel32!BaseThreadInitThunk 2e ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 2nd dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!ZwOpenKeyEx 01 kernel32!LocalBaseRegOpenKey 02 kernel32!RegOpenKeyExInternalW 03 kernel32!RegOpenKeyExW 04 rpcrt4!Server2003NegotiateDisable 05 rpcrt4!IsBindTimeFeatureNegotiationDisabled 06 rpcrt4!OSF_CCONNECTION::SendBindPacket 07 rpcrt4!OSF_CCONNECTION::ActuallyDoBinding 08 rpcrt4!OSF_CCONNECTION::OpenConnectionAndBind 09 rpcrt4!OSF_CCALL::BindToServer 0a rpcrt4!OSF_BINDING_HANDLE::InitCCallWithAssociation 0b rpcrt4!OSF_BINDING_HANDLE::AllocateCCall 0c rpcrt4!OSF_BINDING_HANDLE::NegotiateTransferSyntax 0d rpcrt4!I_RpcNegotiateTransferSyntax 0e rpcrt4!Ndr64pClientSetupTransferSyntax 0f rpcrt4!NdrpClientCall3 10 rpcrt4!NdrClientCall3 11 srvcli!NetShareEnum 12 ntshrui!CShareCache::RefreshNoCritSec 13 ntshrui!CShareCache::Refresh 14 ntshrui!DllMain 15 ntdll!RtlRunOnceExecuteOnce 16 kernel32!InitOnceExecuteOnce 17 ntshrui!DllGetClassObject 18 ole32!CClassCache::CDllPathEntry::DllGetClassObject 19 ole32!CClassCache::CDllFnPtrMoniker::BindToObjectNoSwitch 1a ole32!CClassCache::GetClassObject 1b ole32!CServerContextActivator::CreateInstance 1c ole32!ActivationPropertiesIn::DelegateCreateInstance 1d ole32!CApartmentActivator::CreateInstance 1e ole32!CProcessActivator::CCICallback 1f ole32!CProcessActivator::AttemptActivation 20 ole32!CProcessActivator::ActivateByContext 21 ole32!CProcessActivator::CreateInstance 22 ole32!ActivationPropertiesIn::DelegateCreateInstance 23 ole32!CClientContextActivator::CreateInstance 24 ole32!ActivationPropertiesIn::DelegateCreateInstance 25 ole32!ICoCreateInstanceEx 26 ole32!CoCreateInstance 27 shell32!_SHCoCreateInstance 28 shell32!SHExtCoCreateInstance 29 shell32!DCA_SHExtCoCreateInstance 2a shell32!CFSIconOverlayManager::_s_LoadIconOverlayIdentifiers 2b shell32!CFSIconOverlayManager::CreateInstance 2c shell32!IconOverlayManagerInit 2d shell32!GetIconOverlayManager 2e shell32!FileIconInit 2f shell32!Shell_GetImageLists 30 comdlg32!CPrintDialog::CPrintDialog 31 comdlg32!Print_GeneralDlgProc 32 user32!UserCallDlgProcCheckWow 33 user32!DefDlgProcWorker 34 user32!InternalCreateDialog 35 user32!CreateDialogIndirectParamAorW 36 user32!CreateDialogIndirectParamW 37 comctl32!_CreatePageDialog 38 comctl32!_CreatePage 39 comctl32!PageChange 3a comctl32!InitPropSheetDlg 3b comctl32!PropSheetDlgProc 3c user32!UserCallDlgProcCheckWow 3d user32!DefDlgProcWorker 3e user32!InternalCreateDialog 3f user32!CreateDialogIndirectParamAorW 40 user32!CreateDialogIndirectParamW 41 comctl32!_RealPropertySheet 42 comctl32!_PropertySheet 43 comdlg32!Print_InvokePropertySheets 44 comdlg32!PrintDlgExX 45 comdlg32!PrintDlgExW 46 notepad!GetPrinterDCviaDialog 47 notepad!PrintIt 48 notepad!NPCommand 49 notepad!NPWndProc 4a user32!UserCallWinProcCheckWow 4b user32!DispatchMessageWorker 4c notepad!WinMain 4d notepad!DisplayNonGenuineDlgWorker 4e kernel32!BaseThreadInitThunk 4f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 3rd dump: we see 2 threads start at the same time

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!RtlCompareMemoryUlong 01 ntdll!RtlpAllocateHeap 02 ntdll!RtlAllocateHeap 03 ntdll!RtlDebugAllocateHeap 04 ntdll! ?? ::FNODOBFM::`string' 05 ntdll!RtlAllocateHeap 06 ole32!CRpcResolver::GetThreadWinstaDesktop 07 ole32!CRpcResolver::GetConnection 08 ole32!CoInitializeSecurity 09 ole32!InitializeSecurity 0a ole32!ChannelProcessInitialize 0b ole32!CComApartment::InitRemoting 0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0d ole32!CGIPTable::RegisterInterfaceInGlobal 0e shell32!MarshalToGIT 0f shell32!CBrowserProgressAggregator::BeginSession 10 shell32!IUnknown_BeginBrowserProgressSession 11 shell32!CDefView::CreateViewWindow3 12 shell32!CExplorerBrowser::_CreateViewWindow 13 shell32!CExplorerBrowser::_SwitchView 14 shell32!CExplorerBrowser::_BrowseToView 15 shell32!CExplorerBrowser::_BrowseObjectInternal 16 shell32!CExplorerBrowser::_OnBrowseObject 17 shell32!CExplorerBrowser::BrowseObject 18 comdlg32!CPrintDialog::CreatePrintBrowser 19 comdlg32!CPrintDialog::OnInitDialog 1a comdlg32!Print_GeneralDlgProc 1b user32!UserCallDlgProcCheckWow 1c user32!DefDlgProcWorker 1d user32!InternalCreateDialog 1e user32!CreateDialogIndirectParamAorW 1f user32!CreateDialogIndirectParamW 20 comctl32!_CreatePageDialog 21 comctl32!_CreatePage 22 comctl32!PageChange 23 comctl32!InitPropSheetDlg 24 comctl32!PropSheetDlgProc 25 user32!UserCallDlgProcCheckWow 26 user32!DefDlgProcWorker 27 user32!InternalCreateDialog 28 user32!CreateDialogIndirectParamAorW 29 user32!CreateDialogIndirectParamW 2a comctl32!_RealPropertySheet 2b comctl32!_PropertySheet 2c comdlg32!Print_InvokePropertySheets 2d comdlg32!PrintDlgExX 2e comdlg32!PrintDlgExW 2f notepad!GetPrinterDCviaDialog 30 notepad!PrintIt 31 notepad!NPCommand 32 notepad!NPWndProc 33 user32!UserCallWinProcCheckWow 34 user32!DispatchMessageWorker 35 notepad!WinMain 36 notepad!DisplayNonGenuineDlgWorker 37 kernel32!BaseThreadInitThunk 38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupOpenLocalPolicy 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!RpcBindingSetAuthInfoExW 0c winspool!STRING_HANDLE_bind 0d rpcrt4!GenericHandleMgr 0e rpcrt4!ExplicitBindHandleMgr 0f rpcrt4!Ndr64pClientSetupTransferSyntax 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 winspool!RpcSplOpenPrinter 13 winspool!OpenPrinterRPC 14 winspool!OpenPrinter2W 15 prncache!PrintCache::Listeners::Listener::Start 16 prncache!PrintCache::Listeners::Listener::StartCB 17 ntdll!TppWorkpExecuteCallback 18 ntdll!TppWorkerThread 19 kernel32!BaseThreadInitThunk 1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 0 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 4th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!RtlCompareMemoryUlong 01 ntdll!RtlpAllocateHeap 02 ntdll!RtlAllocateHeap 03 ntdll!RtlDebugAllocateHeap 04 ntdll! ?? ::FNODOBFM::`string' 05 ntdll!RtlAllocateHeap 06 ole32!CRpcResolver::GetThreadWinstaDesktop 07 ole32!CRpcResolver::GetConnection 08 ole32!CoInitializeSecurity 09 ole32!InitializeSecurity 0a ole32!ChannelProcessInitialize 0b ole32!CComApartment::InitRemoting 0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0d ole32!CGIPTable::RegisterInterfaceInGlobal 0e shell32!MarshalToGIT 0f shell32!CBrowserProgressAggregator::BeginSession 10 shell32!IUnknown_BeginBrowserProgressSession 11 shell32!CDefView::CreateViewWindow3 12 shell32!CExplorerBrowser::_CreateViewWindow 13 shell32!CExplorerBrowser::_SwitchView 14 shell32!CExplorerBrowser::_BrowseToView 15 shell32!CExplorerBrowser::_BrowseObjectInternal 16 shell32!CExplorerBrowser::_OnBrowseObject 17 shell32!CExplorerBrowser::BrowseObject 18 comdlg32!CPrintDialog::CreatePrintBrowser 19 comdlg32!CPrintDialog::OnInitDialog 1a comdlg32!Print_GeneralDlgProc 1b user32!UserCallDlgProcCheckWow 1c user32!DefDlgProcWorker 1d user32!InternalCreateDialog 1e user32!CreateDialogIndirectParamAorW 1f user32!CreateDialogIndirectParamW 20 comctl32!_CreatePageDialog 21 comctl32!_CreatePage 22 comctl32!PageChange 23 comctl32!InitPropSheetDlg 24 comctl32!PropSheetDlgProc 25 user32!UserCallDlgProcCheckWow 26 user32!DefDlgProcWorker 27 user32!InternalCreateDialog 28 user32!CreateDialogIndirectParamAorW 29 user32!CreateDialogIndirectParamW 2a comctl32!_RealPropertySheet 2b comctl32!_PropertySheet 2c comdlg32!Print_InvokePropertySheets 2d comdlg32!PrintDlgExX 2e comdlg32!PrintDlgExW 2f notepad!GetPrinterDCviaDialog 30 notepad!PrintIt 31 notepad!NPCommand 32 notepad!NPWndProc 33 user32!UserCallWinProcCheckWow 34 user32!DispatchMessageWorker 35 notepad!WinMain 36 notepad!DisplayNonGenuineDlgWorker 37 kernel32!BaseThreadInitThunk 38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupOpenLocalPolicy 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!RpcBindingSetAuthInfoExW 0c winspool!STRING_HANDLE_bind 0d rpcrt4!GenericHandleMgr 0e rpcrt4!ExplicitBindHandleMgr 0f rpcrt4!Ndr64pClientSetupTransferSyntax 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 winspool!RpcSplOpenPrinter 13 winspool!OpenPrinterRPC 14 winspool!OpenPrinter2W 15 prncache!PrintCache::Listeners::Listener::Start 16 prncache!PrintCache::Listeners::Listener::StartCB 17 ntdll!TppWorkpExecuteCallback 18 ntdll!TppWorkerThread 19 kernel32!BaseThreadInitThunk 1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!LdrpInitializeThread 01 ntdll!LdrpInitialize 02 ntdll!LdrInitializeThunk

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 5th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupClose 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_FAST_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!LRPC_FAST_BINDING_HANDLE::LRPC_FAST_BINDING_HANDLE 0c rpcrt4!LrpcCreateFastBindingHandle 0d rpcrt4!RpcBindingCreateW 0e ole32!CFastBH::CreateFromBindingString 0f ole32!CFastBH::GetOrCreate 10 ole32!CRpcResolver::GetConnection 11 ole32!CoInitializeSecurity 12 ole32!InitializeSecurity 13 ole32!ChannelProcessInitialize 14 ole32!CComApartment::InitRemoting 15 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 16 ole32!CGIPTable::RegisterInterfaceInGlobal 17 shell32!MarshalToGIT 18 shell32!CBrowserProgressAggregator::BeginSession 19 shell32!IUnknown_BeginBrowserProgressSession 1a shell32!CDefView::CreateViewWindow3 1b shell32!CExplorerBrowser::_CreateViewWindow 1c shell32!CExplorerBrowser::_SwitchView 1d shell32!CExplorerBrowser::_BrowseToView 1e shell32!CExplorerBrowser::_BrowseObjectInternal 1f shell32!CExplorerBrowser::_OnBrowseObject 20 shell32!CExplorerBrowser::BrowseObject 21 comdlg32!CPrintDialog::CreatePrintBrowser 22 comdlg32!CPrintDialog::OnInitDialog 23 comdlg32!Print_GeneralDlgProc 24 user32!UserCallDlgProcCheckWow 25 user32!DefDlgProcWorker 26 user32!InternalCreateDialog 27 user32!CreateDialogIndirectParamAorW 28 user32!CreateDialogIndirectParamW 29 comctl32!_CreatePageDialog 2a comctl32!_CreatePage 2b comctl32!PageChange 2c comctl32!InitPropSheetDlg 2d comctl32!PropSheetDlgProc 2e user32!UserCallDlgProcCheckWow 2f user32!DefDlgProcWorker 30 user32!InternalCreateDialog 31 user32!CreateDialogIndirectParamAorW 32 user32!CreateDialogIndirectParamW 33 comctl32!_RealPropertySheet 34 comctl32!_PropertySheet 35 comdlg32!Print_InvokePropertySheets 36 comdlg32!PrintDlgExX 37 comdlg32!PrintDlgExW 38 notepad!GetPrinterDCviaDialog 39 notepad!PrintIt 3a notepad!NPCommand 3b notepad!NPWndProc 3c user32!UserCallWinProcCheckWow 3d user32!DispatchMessageWorker 3e notepad!WinMain 3f notepad!DisplayNonGenuineDlgWorker 40 kernel32!BaseThreadInitThunk 41 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcSplOpenPrinter 05 winspool!OpenPrinterRPC 06 winspool!OpenPrinter2W 07 prncache!PrintCache::Listeners::Listener::Start 08 prncache!PrintCache::Listeners::Listener::StartCB 09 ntdll!TppWorkpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcEnumPrinters 05 winspool!EnumPrintersW 06 prncache!PrintCache::Listeners::ConnectionListener::EnumConnectionsAndRegister 07 prncache!PrintCache::Listeners::ConnectionListener::UpdateCB 08 ntdll!TppWorkpExecuteCallback 09 ntdll!TppWorkerThread 0a kernel32!BaseThreadInitThunk 0b ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 6th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!NtAlpcConnectPort 01 rpcrt4!LRPC_CASSOCIATION::AlpcConnect 02 rpcrt4!LRPC_CASSOCIATION::Connect 03 rpcrt4!LRPC_BASE_BINDING_HANDLE::DriveStateForward 04 rpcrt4!LRPC_FAST_BINDING_HANDLE::Bind 05 rpcrt4!RpcBindingBind 06 ole32!CFastBH::CreateFromBindingString 07 ole32!CFastBH::GetOrCreate 08 ole32!CRpcResolver::GetConnection 09 ole32!CoInitializeSecurity 0a ole32!InitializeSecurity 0b ole32!ChannelProcessInitialize 0c ole32!CComApartment::InitRemoting 0d ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0e ole32!CGIPTable::RegisterInterfaceInGlobal 0f shell32!MarshalToGIT 10 shell32!CBrowserProgressAggregator::BeginSession 11 shell32!IUnknown_BeginBrowserProgressSession 12 shell32!CDefView::CreateViewWindow3 13 shell32!CExplorerBrowser::_CreateViewWindow 14 shell32!CExplorerBrowser::_SwitchView 15 shell32!CExplorerBrowser::_BrowseToView 16 shell32!CExplorerBrowser::_BrowseObjectInternal 17 shell32!CExplorerBrowser::_OnBrowseObject 18 shell32!CExplorerBrowser::BrowseObject 19 comdlg32!CPrintDialog::CreatePrintBrowser 1a comdlg32!CPrintDialog::OnInitDialog 1b comdlg32!Print_GeneralDlgProc 1c user32!UserCallDlgProcCheckWow 1d user32!DefDlgProcWorker 1e user32!InternalCreateDialog 1f user32!CreateDialogIndirectParamAorW 20 user32!CreateDialogIndirectParamW 21 comctl32!_CreatePageDialog 22 comctl32!_CreatePage 23 comctl32!PageChange 24 comctl32!InitPropSheetDlg 25 comctl32!PropSheetDlgProc 26 user32!UserCallDlgProcCheckWow 27 user32!DefDlgProcWorker 28 user32!InternalCreateDialog 29 user32!CreateDialogIndirectParamAorW 2a user32!CreateDialogIndirectParamW 2b comctl32!_RealPropertySheet 2c comctl32!_PropertySheet 2d comdlg32!Print_InvokePropertySheets 2e comdlg32!PrintDlgExX 2f comdlg32!PrintDlgExW 30 notepad!GetPrinterDCviaDialog 31 notepad!PrintIt 32 notepad!NPCommand 33 notepad!NPWndProc 34 user32!UserCallWinProcCheckWow 35 user32!DispatchMessageWorker 36 notepad!WinMain 37 notepad!DisplayNonGenuineDlgWorker 38 kernel32!BaseThreadInitThunk 39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!RtlEnterCriticalSection 01 ntdll!RtlDebugAllocateHeap 02 ntdll! ?? ::FNODOBFM::`string' 03 ntdll!RtlAllocateHeap 04 rpcrt4!AllocWrapper 05 rpcrt4!SID_CACHE::Query 06 rpcrt4!RpcpLookupAccountName 07 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 08 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 09 rpcrt4!RpcBindingSetAuthInfoExW 0a winspool!STRING_HANDLE_bind 0b rpcrt4!GenericHandleMgr 0c rpcrt4!ExplicitBindHandleMgr 0d rpcrt4!Ndr64pClientSetupTransferSyntax 0e rpcrt4!NdrpClientCall3 0f rpcrt4!NdrClientCall3 10 winspool!RpcSplOpenPrinter 11 winspool!OpenPrinterRPC 12 winspool!OpenPrinter2W 13 prncache!PrintCache::Listeners::Listener::Start 14 prncache!PrintCache::Listeners::Listener::StartCB 15 ntdll!TppWorkpExecuteCallback 16 ntdll!TppWorkerThread 17 kernel32!BaseThreadInitThunk 18 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 user32!NtUserAttachThreadInput 01 shell32!CWaitTask::s_WaitBeforeCursing 02 ntdll!RtlpTpWaitCallback 03 ntdll!TppWaitpExecuteCallback 04 ntdll!TppWorkerThread 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 7th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 rpcrt4!RpcStringFreeW 01 ole32!CFastBH::CreateFromBindingString 02 ole32!CFastBH::GetOrCreate 03 ole32!CRpcResolver::GetConnection 04 ole32!CoInitializeSecurity 05 ole32!InitializeSecurity 06 ole32!ChannelProcessInitialize 07 ole32!CComApartment::InitRemoting 08 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 09 ole32!CGIPTable::RegisterInterfaceInGlobal 0a shell32!MarshalToGIT 0b shell32!CBrowserProgressAggregator::BeginSession 0c shell32!IUnknown_BeginBrowserProgressSession 0d shell32!CDefView::CreateViewWindow3 0e shell32!CExplorerBrowser::_CreateViewWindow 0f shell32!CExplorerBrowser::_SwitchView 10 shell32!CExplorerBrowser::_BrowseToView 11 shell32!CExplorerBrowser::_BrowseObjectInternal 12 shell32!CExplorerBrowser::_OnBrowseObject 13 shell32!CExplorerBrowser::BrowseObject 14 comdlg32!CPrintDialog::CreatePrintBrowser 15 comdlg32!CPrintDialog::OnInitDialog 16 comdlg32!Print_GeneralDlgProc 17 user32!UserCallDlgProcCheckWow 18 user32!DefDlgProcWorker 19 user32!InternalCreateDialog 1a user32!CreateDialogIndirectParamAorW 1b user32!CreateDialogIndirectParamW 1c comctl32!_CreatePageDialog 1d comctl32!_CreatePage 1e comctl32!PageChange 1f comctl32!InitPropSheetDlg 20 comctl32!PropSheetDlgProc 21 user32!UserCallDlgProcCheckWow 22 user32!DefDlgProcWorker 23 user32!InternalCreateDialog 24 user32!CreateDialogIndirectParamAorW 25 user32!CreateDialogIndirectParamW 26 comctl32!_RealPropertySheet 27 comctl32!_PropertySheet 28 comdlg32!Print_InvokePropertySheets 29 comdlg32!PrintDlgExX 2a comdlg32!PrintDlgExW 2b notepad!GetPrinterDCviaDialog 2c notepad!PrintIt 2d notepad!NPCommand 2e notepad!NPWndProc 2f user32!UserCallWinProcCheckWow 30 user32!DispatchMessageWorker 31 notepad!WinMain 32 notepad!DisplayNonGenuineDlgWorker 33 kernel32!BaseThreadInitThunk 34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcSplOpenPrinter 05 winspool!OpenPrinterRPC 06 winspool!OpenPrinter2W 07 prncache!PrintCache::Listeners::Listener::Start 08 prncache!PrintCache::Listeners::Listener::StartCB 09 ntdll!TppWorkpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 8th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 user32!NtUserSetWindowLongPtr 01 user32!SetWindowLongPtr 02 ole32!OXIDEntry::StartServer 03 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 04 ole32!CGIPTable::RegisterInterfaceInGlobal 05 shell32!MarshalToGIT 06 shell32!CBrowserProgressAggregator::BeginSession 07 shell32!IUnknown_BeginBrowserProgressSession 08 shell32!CDefView::CreateViewWindow3 09 shell32!CExplorerBrowser::_CreateViewWindow 0a shell32!CExplorerBrowser::_SwitchView 0b shell32!CExplorerBrowser::_BrowseToView 0c shell32!CExplorerBrowser::_BrowseObjectInternal 0d shell32!CExplorerBrowser::_OnBrowseObject 0e shell32!CExplorerBrowser::BrowseObject 0f comdlg32!CPrintDialog::CreatePrintBrowser 10 comdlg32!CPrintDialog::OnInitDialog 11 comdlg32!Print_GeneralDlgProc 12 user32!UserCallDlgProcCheckWow 13 user32!DefDlgProcWorker 14 user32!InternalCreateDialog 15 user32!CreateDialogIndirectParamAorW 16 user32!CreateDialogIndirectParamW 17 comctl32!_CreatePageDialog 18 comctl32!_CreatePage 19 comctl32!PageChange 1a comctl32!InitPropSheetDlg 1b comctl32!PropSheetDlgProc 1c user32!UserCallDlgProcCheckWow 1d user32!DefDlgProcWorker 1e user32!InternalCreateDialog 1f user32!CreateDialogIndirectParamAorW 20 user32!CreateDialogIndirectParamW 21 comctl32!_RealPropertySheet 22 comctl32!_PropertySheet 23 comdlg32!Print_InvokePropertySheets 24 comdlg32!PrintDlgExX 25 comdlg32!PrintDlgExW 26 notepad!GetPrinterDCviaDialog 27 notepad!PrintIt 28 notepad!NPCommand 29 notepad!NPWndProc 2a user32!UserCallWinProcCheckWow 2b user32!DispatchMessageWorker 2c notepad!WinMain 2d notepad!DisplayNonGenuineDlgWorker 2e kernel32!BaseThreadInitThunk 2f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 9th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 user32!NtUserPeekMessage 01 user32!PeekMessageW 02 shell32!PeekMessageWithWakeMask 03 shell32!SHProcessMessagesUntilEventsEx 04 shell32!CDefView::_SetItemCollection 05 shell32!CDefView::_CreateNewCollection 06 shell32!CDefView::CreateViewWindow3 07 shell32!CExplorerBrowser::_CreateViewWindow 08 shell32!CExplorerBrowser::_SwitchView 09 shell32!CExplorerBrowser::_BrowseToView 0a shell32!CExplorerBrowser::_BrowseObjectInternal 0b shell32!CExplorerBrowser::_OnBrowseObject 0c shell32!CExplorerBrowser::BrowseObject 0d comdlg32!CPrintDialog::CreatePrintBrowser 0e comdlg32!CPrintDialog::OnInitDialog 0f comdlg32!Print_GeneralDlgProc 10 user32!UserCallDlgProcCheckWow 11 user32!DefDlgProcWorker 12 user32!InternalCreateDialog 13 user32!CreateDialogIndirectParamAorW 14 user32!CreateDialogIndirectParamW 15 comctl32!_CreatePageDialog 16 comctl32!_CreatePage 17 comctl32!PageChange 18 comctl32!InitPropSheetDlg 19 comctl32!PropSheetDlgProc 1a user32!UserCallDlgProcCheckWow 1b user32!DefDlgProcWorker 1c user32!InternalCreateDialog 1d user32!CreateDialogIndirectParamAorW 1e user32!CreateDialogIndirectParamW 1f comctl32!_RealPropertySheet 20 comctl32!_PropertySheet 21 comdlg32!Print_InvokePropertySheets 22 comdlg32!PrintDlgExX 23 comdlg32!PrintDlgExW 24 notepad!GetPrinterDCviaDialog 25 notepad!PrintIt 26 notepad!NPCommand 27 notepad!NPWndProc 28 user32!UserCallWinProcCheckWow 29 user32!DispatchMessageWorker 2a notepad!WinMain 2b notepad!DisplayNonGenuineDlgWorker 2c kernel32!BaseThreadInitThunk 2d ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 KERNELBASE!WaitForSingleObjectEx 02 shlwapi!CreateThreadWorker 03 shlwapi!SHCreateThread 04 shell32!CEnumThread::Run 05 shell32!CEnumTask::_StartEnumThread 06 shell32!CEnumTask::_IncrEnumFolder 07 shell32!CEnumTask::InternalResumeRT 08 shell32!CRunnableTask::Run 09 shell32!CShellTask::TT_Run 0a shell32!CShellTaskThread::ThreadProc 0b shell32!CShellTaskThread::s_ThreadProc 0c shlwapi!ExecuteWorkItemThreadProc 0d ntdll!RtlpTpWorkCallback 0e ntdll!TppWorkerThread 0f kernel32!BaseThreadInitThunk 10 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.1310 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 10th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 kernel32!TlsGetValue 01 usp10!UspFreeMem 02 usp10!ScriptApplyDigitSubstitution 03 lpk!LpkCharsetDraw 04 lpk!LpkDrawTextEx 05 user32!DT_GetExtentMinusPrefixes 06 user32!NeedsEndEllipsis 07 user32!AddEllipsisAndDrawLine 08 user32!DrawTextExWorker 09 user32!DrawTextW 0a comctl32!CLVView::_ComputeLabelSizeWorker 0b comctl32!CLVView::v_RecomputeLabelSize 0c comctl32!CLVListView::v_DrawItem 0d comctl32!CLVDrawItemManager::DrawItem 0e comctl32!CLVDrawManager::_PaintItems 0f comctl32!CLVDrawManager::_PaintWorkArea 10 comctl32!CLVDrawManager::_OnPaintWorkAreas 11 comctl32!CLVDrawManager::_OnPaint 12 comctl32!CListView::WndProc 13 comctl32!CListView::s_WndProc 14 user32!UserCallWinProcCheckWow 15 user32!CallWindowProcAorW 16 user32!CallWindowProcW 17 comctl32!CallOriginalWndProc 18 comctl32!CallNextSubclassProc 19 comctl32!DefSubclassProc 1a shell32!DefSubclassProc 1b shell32!CListViewHost::s_ListViewSubclassWndProc 1c comctl32!CallNextSubclassProc 1d comctl32!MasterSubclassProc 1e user32!UserCallWinProcCheckWow 1f user32!DispatchClientMessage 20 user32!_fnDWORD 21 ntdll!KiUserCallbackDispatcherContinue 22 user32!NtUserDispatchMessage 23 user32!DispatchMessageWorker 24 user32!IsDialogMessageW 25 comctl32!Prop_IsDialogMessage 26 comctl32!_RealPropertySheet 27 comctl32!_PropertySheet 28 comdlg32!Print_InvokePropertySheets 29 comdlg32!PrintDlgExX 2a comdlg32!PrintDlgExW 2b notepad!GetPrinterDCviaDialog 2c notepad!PrintIt 2d notepad!NPCommand 2e notepad!NPWndProc 2f user32!UserCallWinProcCheckWow 30 user32!DispatchMessageWorker 31 notepad!WinMain 32 notepad!DisplayNonGenuineDlgWorker 33 kernel32!BaseThreadInitThunk 34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtQueryKey 01 kernel32!BaseRegGetKeySemantics 02 kernel32!BaseRegGetUserAndMachineClass 03 kernel32!LocalBaseRegQueryValue 04 kernel32!RegQueryValueExW 05 shlwapi!SHRegQueryValueW 06 shlwapi!SHRegGetValueW 07 shlwapi!SHQueryValueExW 08 shell32!_GetServerInfo 09 shell32!_SHCoCreateInstance 0a shell32!CRegFolder::_CreateCachedRegFolder 0b shell32!CRegFolder::_BindToItem 0c shell32!CRegFolder::BindToObject 0d shell32!CRegFolder::_BindToItem 0e shell32!CRegFolder::BindToObject 0f shell32!SHBindToObject 10 shell32!CIconOverlayTask::InternalResumeRT 11 shell32!CRunnableTask::Run 12 shell32!CShellTask::TT_Run 13 shell32!CShellTaskThread::ThreadProc 14 shell32!CShellTaskThread::s_ThreadProc 15 shlwapi!ExecuteWorkItemThreadProc 16 ntdll!RtlpTpWorkCallback 17 ntdll!TppWorkerThread 18 kernel32!BaseThreadInitThunk 19 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 11th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 usp10!GenericEngineGetGlyphs 01 usp10!ShlShape 02 usp10!ScriptShape 03 usp10!RenderItemNoFallback 04 usp10!RenderItemWithFallback 05 usp10!RenderItem 06 usp10!ScriptStringAnalyzeGlyphs 07 usp10!ScriptStringAnalyse 08 lpk!LpkCharsetDraw 09 lpk!LpkDrawTextEx 0a user32!DT_GetExtentMinusPrefixes 0b user32!NeedsEndEllipsis 0c user32!AddEllipsisAndDrawLine 0d user32!DrawTextExWorker 0e user32!DrawTextW 0f comctl32!CLVView::_ComputeLabelSizeWorker 10 comctl32!CLVView::v_RecomputeLabelSize 11 comctl32!CLVListView::v_DrawItem 12 comctl32!CLVDrawItemManager::DrawItem 13 comctl32!CLVDrawManager::_PaintItems 14 comctl32!CLVDrawManager::_PaintWorkArea 15 comctl32!CLVDrawManager::_OnPaintWorkAreas 16 comctl32!CLVDrawManager::_OnPaint 17 comctl32!CListView::WndProc 18 comctl32!CListView::s_WndProc 19 user32!UserCallWinProcCheckWow 1a user32!CallWindowProcAorW 1b user32!CallWindowProcW 1c comctl32!CallOriginalWndProc 1d comctl32!CallNextSubclassProc 1e comctl32!DefSubclassProc 1f shell32!DefSubclassProc 20 shell32!CListViewHost::s_ListViewSubclassWndProc 21 comctl32!CallNextSubclassProc 22 comctl32!MasterSubclassProc 23 user32!UserCallWinProcCheckWow 24 user32!DispatchClientMessage 25 user32!_fnDWORD 26 ntdll!KiUserCallbackDispatcherContinue 27 user32!NtUserDispatchMessage 28 user32!DispatchMessageWorker 29 user32!IsDialogMessageW 2a comctl32!Prop_IsDialogMessage 2b comctl32!_RealPropertySheet 2c comctl32!_PropertySheet 2d comdlg32!Print_InvokePropertySheets 2e comdlg32!PrintDlgExX 2f comdlg32!PrintDlgExW 30 notepad!GetPrinterDCviaDialog 31 notepad!PrintIt 32 notepad!NPCommand 33 notepad!NPWndProc 34 user32!UserCallWinProcCheckWow 35 user32!DispatchMessageWorker 36 notepad!WinMain 37 notepad!DisplayNonGenuineDlgWorker 38 kernel32!BaseThreadInitThunk 39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 ntdll!RtlpWaitOnCriticalSection 02 ntdll!RtlEnterCriticalSection 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 ntdll!RtlpWaitOnCriticalSection 02 ntdll!RtlEnterCriticalSection 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 user32!NtUserGetIconInfo 01 user32!CopyIcoCur 02 user32!InternalCopyImage 03 user32!CopyImage 04 comctl32!CImageList::_ReplaceIcon 05 comctl32!CImageList::ReplaceIcon 06 comctl32!CSparseImageList::ReplaceIcon 07 shell32!CIconCache::AddToBackIconTable 08 shell32!AddToBackIconTable 09 shell32!SHAddIconsToCache 0a shell32!_GetILIndexGivenPXIcon 0b shell32!_GetILIndexFromItem 0c shell32!SHGetIconIndexFromPIDL 0d shell32!MapIDListToIconILIndex 0e shell32!CLoadSystemIconTask::InternalResumeRT 0f shell32!CRunnableTask::Run 10 shell32!CShellTask::TT_Run 11 shell32!CShellTaskThread::ThreadProc 12 shell32!CShellTaskThread::s_ThreadProc 13 shlwapi!ExecuteWorkItemThreadProc 14 ntdll!RtlpTpWorkCallback 15 ntdll!TppWorkerThread 16 kernel32!BaseThreadInitThunk 17 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 gdi32!NtUserSelectPalette 01 gdi32!SelectPalette 02 gdi32!SetDIBits 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcFindNextPrinterChangeNotification 05 winspool!FindNextPrinterChangeNotification 06 prncache!PrintCache::Listeners::Listener::ProcessWait 07 prncache!PrintCache::Listeners::Listener::ProcessWaitCB 08 ntdll!TppWaitpExecuteCallback 09 ntdll!TppWorkerThread 0a kernel32!BaseThreadInitThunk 0b ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 12th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 usp10!otlChainingLookup::apply 01 usp10!ApplyLookup 02 usp10!ApplyFeatures 03 usp10!SubstituteOtlGlyphs 04 usp10!GenericEngineGetGlyphs 05 usp10!ShlShape 06 usp10!ScriptShape 07 usp10!RenderItemNoFallback 08 usp10!RenderItemWithFallback 09 usp10!RenderItem 0a usp10!ScriptStringAnalyzeGlyphs 0b usp10!ScriptStringAnalyse 0c lpk!LpkCharsetDraw 0d lpk!LpkDrawTextEx 0e user32!DT_DrawStr 0f user32!DT_DrawJustifiedLine 10 user32!DrawTextExWorker 11 user32!DrawTextW 12 comctl32!CLVView::_ComputeLabelSizeWorker 13 comctl32!CLVView::v_RecomputeLabelSize 14 comctl32!CLVListView::v_DrawItem 15 comctl32!CLVDrawItemManager::DrawItem 16 comctl32!CLVDrawManager::_PaintItems 17 comctl32!CLVDrawManager::_PaintWorkArea 18 comctl32!CLVDrawManager::_OnPaintWorkAreas 19 comctl32!CLVDrawManager::_OnPaint 1a comctl32!CListView::WndProc 1b comctl32!CListView::s_WndProc 1c user32!UserCallWinProcCheckWow 1d user32!CallWindowProcAorW 1e user32!CallWindowProcW 1f comctl32!CallOriginalWndProc 20 comctl32!CallNextSubclassProc 21 comctl32!DefSubclassProc 22 shell32!DefSubclassProc 23 shell32!CListViewHost::s_ListViewSubclassWndProc 24 comctl32!CallNextSubclassProc 25 comctl32!MasterSubclassProc 26 user32!UserCallWinProcCheckWow 27 user32!DispatchClientMessage 28 user32!_fnDWORD 29 ntdll!KiUserCallbackDispatcherContinue 2a user32!NtUserDispatchMessage 2b user32!DispatchMessageWorker 2c user32!IsDialogMessageW 2d comctl32!Prop_IsDialogMessage 2e comctl32!_RealPropertySheet 2f comctl32!_PropertySheet 30 comdlg32!Print_InvokePropertySheets 31 comdlg32!PrintDlgExX 32 comdlg32!PrintDlgExW 33 notepad!GetPrinterDCviaDialog 34 notepad!PrintIt 35 notepad!NPCommand 36 notepad!NPWndProc 37 user32!UserCallWinProcCheckWow 38 user32!DispatchMessageWorker 39 notepad!WinMain 3a notepad!DisplayNonGenuineDlgWorker 3b kernel32!BaseThreadInitThunk 3c ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtUnmapViewOfSection 01 KERNELBASE!FreeLibrary 02 user32!PrivateExtractIconsW 03 shell32!SHPrivateExtractIcons 04 shell32!SHDefExtractIconW 05 shell32!CExtractIcon::_ExtractW 06 shell32!CExtractIconBase::Extract 07 shell32!IExtractIcon_Extract 08 shell32!_GetILIndexGivenPXIcon 09 shell32!_GetILIndexFromItem 0a shell32!SHGetIconIndexFromPIDL 0b shell32!MapIDListToIconILIndex 0c shell32!CLoadSystemIconTask::InternalResumeRT 0d shell32!CRunnableTask::Run 0e shell32!CShellTask::TT_Run 0f shell32!CShellTaskThread::ThreadProc 10 shell32!CShellTaskThread::s_ThreadProc 11 shlwapi!ExecuteWorkItemThreadProc 12 ntdll!RtlpTpWorkCallback 13 ntdll!TppWorkerThread 14 kernel32!BaseThreadInitThunk 15 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 gdi32!ZwGdiSetDIBitsToDeviceInternal 01 gdi32!SetDIBitsToDevice 02 gdi32!SetDIBits 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!ZwMapViewOfSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CASSOCIATION::AlpcSendWaitReceivePort 02 rpcrt4!LRPC_BASE_CCALL::DoSendReceive 03 rpcrt4!LRPC_BASE_CCALL::SendReceive 04 rpcrt4!NdrpClientCall2 05 rpcrt4!NdrClientCall2 06 ole32!CRpcResolver::BulkUpdateOIDs 07 ole32!CROIDTable::ClientBulkUpdateOIDWithPingServer 08 ole32!CROIDTable::WorkerThreadLoop 09 ole32!CRpcThread::WorkerLoop 0a ole32!CRpcThreadCache::RpcWorkerThreadEntry 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

11 Id: 1344.1fb0 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Software Trace Analysis, Trace Analysis Patterns, x64 Windows | No Comments »

Spiking Thread, Top Module, Module Hint, and Memory Fluctuation: pattern cooperation

Saturday, October 24th, 2015

We noticed that after restarting Windows 7 system on one of our notebooks it becomes very sluggish. Task Manager showed 25% CPU usage in one of svchost.exe processes and very high usage of physical memory. So we immediately dumped it using procdump. The resulted process memory dump was almost 1.5 GB. Although the analysis case is very simple and straightforward we decided to publish to show the value of crash and hang dump analysis in understanding abnormal software behavior in “user” context.

When we open the memory dump and run !runaway WinDbg command we immediately recognize Spiking Thread pattern:

0:000> !runaway f User Mode Time Thread Time 38:14a0 0 days 0:04:59.911 36:a88 0 days 0:00:00.187 2:41c 0 days 0:00:00.187 39:19f8 0 days 0:00:00.046 34:fa8 0 days 0:00:00.046 21:12b4 0 days 0:00:00.031 43:8f0 0 days 0:00:00.015 42:1504 0 days 0:00:00.015 41:1a20 0 days 0:00:00.015 40:978 0 days 0:00:00.015 33:e0c 0 days 0:00:00.015 32:ff8 0 days 0:00:00.015 22:1304 0 days 0:00:00.015 19:f68 0 days 0:00:00.015 9:664 0 days 0:00:00.015 8:660 0 days 0:00:00.015 6:518 0 days 0:00:00.015 4:4a4 0 days 0:00:00.015 51:160c 0 days 0:00:00.000 50:1590 0 days 0:00:00.000 49:15d8 0 days 0:00:00.000 48:ac8 0 days 0:00:00.000 47:14d0 0 days 0:00:00.000 46:1bfc 0 days 0:00:00.000 45:18e8 0 days 0:00:00.000 44:1448 0 days 0:00:00.000 37:1910 0 days 0:00:00.000 35:558 0 days 0:00:00.000 31:14b8 0 days 0:00:00.000 30:14b4 0 days 0:00:00.000 29:14ac 0 days 0:00:00.000 28:13d0 0 days 0:00:00.000 27:13c8 0 days 0:00:00.000 26:13b4 0 days 0:00:00.000 25:13b0 0 days 0:00:00.000 24:13a8 0 days 0:00:00.000 23:1328 0 days 0:00:00.000 20:12b0 0 days 0:00:00.000 18:e20 0 days 0:00:00.000 17:e10 0 days 0:00:00.000 16:dd0 0 days 0:00:00.000 15:ce0 0 days 0:00:00.000 14:754 0 days 0:00:00.000 13:718 0 days 0:00:00.000 12:678 0 days 0:00:00.000 11:674 0 days 0:00:00.000 10:668 0 days 0:00:00.000 7:548 0 days 0:00:00.000 5:4ac 0 days 0:00:00.000 3:4a0 0 days 0:00:00.000 1:418 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Kernel Mode Time Thread Time 38:14a0 0 days 0:00:55.707 36:a88 0 days 0:00:01.778 2:41c 0 days 0:00:00.405 34:fa8 0 days 0:00:00.109 9:664 0 days 0:00:00.062 43:8f0 0 days 0:00:00.046 42:1504 0 days 0:00:00.046 21:12b4 0 days 0:00:00.046 32:ff8 0 days 0:00:00.031 22:1304 0 days 0:00:00.031 18:e20 0 days 0:00:00.031 39:19f8 0 days 0:00:00.015 19:f68 0 days 0:00:00.015 6:518 0 days 0:00:00.015 3:4a0 0 days 0:00:00.015 1:418 0 days 0:00:00.015 51:160c 0 days 0:00:00.000 50:1590 0 days 0:00:00.000 49:15d8 0 days 0:00:00.000 48:ac8 0 days 0:00:00.000 47:14d0 0 days 0:00:00.000 46:1bfc 0 days 0:00:00.000 45:18e8 0 days 0:00:00.000 44:1448 0 days 0:00:00.000 41:1a20 0 days 0:00:00.000 40:978 0 days 0:00:00.000 37:1910 0 days 0:00:00.000 35:558 0 days 0:00:00.000 33:e0c 0 days 0:00:00.000 31:14b8 0 days 0:00:00.000 30:14b4 0 days 0:00:00.000 29:14ac 0 days 0:00:00.000 28:13d0 0 days 0:00:00.000 27:13c8 0 days 0:00:00.000 26:13b4 0 days 0:00:00.000 25:13b0 0 days 0:00:00.000 24:13a8 0 days 0:00:00.000 23:1328 0 days 0:00:00.000 20:12b0 0 days 0:00:00.000 17:e10 0 days 0:00:00.000 16:dd0 0 days 0:00:00.000 15:ce0 0 days 0:00:00.000 14:754 0 days 0:00:00.000 13:718 0 days 0:00:00.000 12:678 0 days 0:00:00.000 11:674 0 days 0:00:00.000 10:668 0 days 0:00:00.000 8:660 0 days 0:00:00.000 7:548 0 days 0:00:00.000 5:4ac 0 days 0:00:00.000 4:4a4 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Elapsed Time Thread Time 0:410 0 days 0:10:24.550 2:41c 0 days 0:10:24.534 1:418 0 days 0:10:24.534 4:4a4 0 days 0:10:24.331 3:4a0 0 days 0:10:24.331 5:4ac 0 days 0:10:24.269 6:518 0 days 0:10:23.957 7:548 0 days 0:10:23.817 8:660 0 days 0:10:22.176 9:664 0 days 0:10:22.156 10:668 0 days 0:10:22.126 11:674 0 days 0:10:22.026 12:678 0 days 0:10:21.986 13:718 0 days 0:10:20.066 14:754 0 days 0:10:20.056 15:ce0 0 days 0:10:15.131 16:dd0 0 days 0:10:14.641 17:e10 0 days 0:10:14.551 18:e20 0 days 0:10:14.531 19:f68 0 days 0:10:13.611 21:12b4 0 days 0:10:10.647 20:12b0 0 days 0:10:10.647 22:1304 0 days 0:10:10.553 23:1328 0 days 0:10:10.381 24:13a8 0 days 0:10:09.024 26:13b4 0 days 0:10:08.931 25:13b0 0 days 0:10:08.931 28:13d0 0 days 0:10:08.899 27:13c8 0 days 0:10:08.899 31:14b8 0 days 0:10:07.932 30:14b4 0 days 0:10:07.932 29:14ac 0 days 0:10:07.932 32:ff8 0 days 0:08:11.785 33:e0c 0 days 0:08:11.644 34:fa8 0 days 0:08:06.750 35:558 0 days 0:08:05.765 36:a88 0 days 0:08:05.127 37:1910 0 days 0:08:02.608 38:14a0 0 days 0:07:19.276 42:1504 0 days 0:04:55.634 41:1a20 0 days 0:04:55.634 40:978 0 days 0:04:55.634 39:19f8 0 days 0:04:55.634 43:8f0 0 days 0:04:55.618 44:1448 0 days 0:04:42.634 46:1bfc 0 days 0:04:20.945 45:18e8 0 days 0:04:20.945 47:14d0 0 days 0:02:42.515 48:ac8 0 days 0:01:27.434 50:1590 0 days 0:00:04.917 49:15d8 0 days 0:00:04.917 51:160c 0 days 0:00:03.387

We see that most of the time the thread was consuming CPU in user mode and less in kernel mode.

We then examine that thread stack trace:

0:000> ~38k # Child-SP RetAddr Call Site 00 00000000`0414ddd8 00000000`7769cf66 ntdll!ZwQueryPerformanceCounter+0xa *** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll - 01 00000000`0414dde0 000007fe`ee7ee7aa ntdll!RtlQueryPerformanceFrequency+0x16 02 00000000`0414de10 000007fe`ee7ee53f wuaueng!DllInstall+0×153da 03 00000000`0414de40 000007fe`ee7e0d99 wuaueng!DllInstall+0×1516f 04 00000000`0414e190 000007fe`ee7df542 wuaueng!DllInstall+0×79c9 05 00000000`0414e1f0 000007fe`ee7df57c wuaueng!DllInstall+0×6172 06 00000000`0414e2b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 07 00000000`0414e370 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 08 00000000`0414e430 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 09 00000000`0414e4f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0a 00000000`0414e5b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0b 00000000`0414e670 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0c 00000000`0414e730 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0d 00000000`0414e7f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0e 00000000`0414e8b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0f 00000000`0414e970 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 10 00000000`0414ea30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 11 00000000`0414eaf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 12 00000000`0414ebb0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 13 00000000`0414ec70 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 14 00000000`0414ed30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 15 00000000`0414edf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 16 00000000`0414eeb0 000007fe`ee7df2d7 wuaueng!DllInstall+0×61ac 17 00000000`0414ef70 000007fe`ee7e9b4f wuaueng!DllInstall+0×5f07 18 00000000`0414f020 000007fe`ee7eb7e8 wuaueng!DllInstall+0×1077f 19 00000000`0414f370 000007fe`ee8010b2 wuaueng!DllInstall+0×12418 1a 00000000`0414f5b0 000007fe`ee7fe53e wuaueng!DllInstall+0×27ce2 1b 00000000`0414f610 000007fe`ee7fccac wuaueng!DllInstall+0×2516e 1c 00000000`0414f660 000007fe`ee7dec19 wuaueng!DllInstall+0×238dc 1d 00000000`0414f690 000007fe`ee7de30f wuaueng!DllInstall+0×5849 1e 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0×4f3f 1f 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd 20 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that Top Module is from Windows Update Agent:

0:000> lmv m wuaueng Browse full module list start end module name 000007fe`ee740000 000007fe`ee9a0000 wuaueng (export symbols) wuaueng.dll Loaded symbol image file: wuaueng.dll Image path: c:\Windows\System32\wuaueng.dll Image name: wuaueng.dll Browse all global symbols functions data Timestamp: Wed May 14 17:21:24 2014 (53739804) CheckSum: 00265DEA ImageSize: 00260000 File version: 7.6.7600.320 Product version: 7.6.7600.320 File flags: 8 (Mask 3F) Private File OS: 40004 NT Win32 File type: 2.0 Dll File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® Windows® Operating System InternalName: wuaueng.dll OriginalFilename: wuaueng.dll ProductVersion: 7.6.7600.320 FileVersion: 7.6.7600.320 (winmain_wtr_wsus3sp2(oobla).140514-0916) FileDescription: Windows Update Agent LegalCopyright: © Microsoft Corporation. All rights reserved.

We also examine process address space:

0:000> !address -summary

Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 507 7ff`9373b000 ( 7.998 TB) 99.98% Heap 574 0`3e2ad000 ( 994.676 MB) 57.27% 0.01% 9786 0`2803e000 ( 640.242 MB) 36.87% 0.01% Image 874 0`049a6000 ( 73.648 MB) 4.24% 0.00% Stack 156 0`01a00000 ( 26.000 MB) 1.50% 0.00% Other 13 0`001bb000 ( 1.730 MB) 0.10% 0.00% TEB 52 0`00068000 ( 416.000 kB) 0.02% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 834 0`5345e000 ( 1.301 GB) 76.72% 0.02% MEM_MAPPED 9747 0`14ab0000 ( 330.688 MB) 19.04% 0.00% MEM_IMAGE 875 0`049a7000 ( 73.652 MB) 4.24% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 507 7ff`9373b000 ( 7.998 TB) 99.98% MEM_COMMIT 11164 0`513d8000 ( 1.269 GB) 74.85% 0.02% MEM_RESERVE 292 0`1b4dd000 ( 436.863 MB) 25.15% 0.01%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READWRITE 5533 0`41d6c000 ( 1.029 GB) 60.66% 0.01% PAGE_WRITECOPY 4911 0`098b9000 ( 152.723 MB) 8.79% 0.00% PAGE_READONLY 504 0`03204000 ( 50.016 MB) 2.88% 0.00% PAGE_EXECUTE_READ 164 0`02b29000 ( 43.160 MB) 2.49% 0.00% PAGE_READWRITE|PAGE_GUARD 52 0`00086000 ( 536.000 kB) 0.03% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 0`ff84b000 7fd`eed65000 ( 7.992 TB) Heap 0`83eb0000 0`01c17000 ( 28.090 MB) 0`93070000 0`0ffbd000 ( 255.738 MB) Image 7fe`fe18a000 0`0089e000 ( 8.617 MB) Stack 0`07140000 0`0007b000 ( 492.000 kB) Other 0`00760000 0`00181000 ( 1.504 MB) TEB 7ff`fff0a000 0`00002000 ( 8.000 kB) PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

We see that process heap occupies almost 1 GB. Let’s see its statistics:

0:000> !heap -s

****************************** NT HEAP STATS BELOW ****************************** LFH Key : 0x000000a57ddeb5ed Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ------------------------------------------------------------------------------------- Virtual block: 0000000051eb0000 - 0000000051eb0000 (size 0000000000000000) Virtual block: 0000000052930000 - 0000000052930000 (size 0000000000000000) Virtual block: 0000000026ba0000 - 0000000026ba0000 (size 0000000000000000) Virtual block: 0000000053e50000 - 0000000053e50000 (size 0000000000000000) Virtual block: 0000000054490000 - 0000000054490000 (size 0000000000000000) Virtual block: 00000000547a0000 - 00000000547a0000 (size 0000000000000000) Virtual block: 000000003bda0000 - 000000003bda0000 (size 0000000000000000) Virtual block: 0000000056030000 - 0000000056030000 (size 0000000000000000) Virtual block: 00000000567c0000 - 00000000567c0000 (size 0000000000000000) Virtual block: 00000000572a0000 - 00000000572a0000 (size 0000000000000000) Virtual block: 0000000057870000 - 0000000057870000 (size 0000000000000000) Virtual block: 0000000045a10000 - 0000000045a10000 (size 0000000000000000) Virtual block: 0000000058fb0000 - 0000000058fb0000 (size 0000000000000000) Virtual block: 0000000045c10000 - 0000000045c10000 (size 0000000000000000) Virtual block: 00000000599c0000 - 00000000599c0000 (size 0000000000000000) Virtual block: 0000000059ff0000 - 0000000059ff0000 (size 0000000000000000) Virtual block: 000000005ae20000 - 000000005ae20000 (size 0000000000000000) Virtual block: 000000005c5d0000 - 000000005c5d0000 (size 0000000000000000) Virtual block: 0000000054b90000 - 0000000054b90000 (size 0000000000000000) Virtual block: 000000005d070000 - 000000005d070000 (size 0000000000000000) Virtual block: 000000006e370000 - 000000006e370000 (size 0000000000000000) Virtual block: 000000006f8e0000 - 000000006f8e0000 (size 0000000000000000) Virtual block: 000000006ed20000 - 000000006ed20000 (size 0000000000000000) Virtual block: 0000000070890000 - 0000000070890000 (size 0000000000000000) Virtual block: 000000005e370000 - 000000005e370000 (size 0000000000000000) Virtual block: 000000005f5a0000 - 000000005f5a0000 (size 0000000000000000) Virtual block: 000000005fa60000 - 000000005fa60000 (size 0000000000000000) Virtual block: 000000005ffe0000 - 000000005ffe0000 (size 0000000000000000) Virtual block: 0000000060770000 - 0000000060770000 (size 0000000000000000) Virtual block: 0000000060aa0000 - 0000000060aa0000 (size 0000000000000000) Virtual block: 0000000061810000 - 0000000061810000 (size 0000000000000000) Virtual block: 0000000061a30000 - 0000000061a30000 (size 0000000000000000) Virtual block: 0000000061f00000 - 0000000061f00000 (size 0000000000000000) Virtual block: 0000000064470000 - 0000000064470000 (size 0000000000000000) Virtual block: 0000000064c00000 - 0000000064c00000 (size 0000000000000000) Virtual block: 00000000656d0000 - 00000000656d0000 (size 0000000000000000) Virtual block: 00000000660f0000 - 00000000660f0000 (size 0000000000000000) Virtual block: 0000000066530000 - 0000000066530000 (size 0000000000000000) Virtual block: 00000000669d0000 - 00000000669d0000 (size 0000000000000000) Virtual block: 00000000676a0000 - 00000000676a0000 (size 0000000000000000) Virtual block: 0000000067a70000 - 0000000067a70000 (size 0000000000000000) Virtual block: 0000000068a20000 - 0000000068a20000 (size 0000000000000000) Virtual block: 0000000069f10000 - 0000000069f10000 (size 0000000000000000) Virtual block: 000000006a6c0000 - 000000006a6c0000 (size 0000000000000000) Virtual block: 000000006ad80000 - 000000006ad80000 (size 0000000000000000) Virtual block: 000000006b9a0000 - 000000006b9a0000 (size 0000000000000000) Virtual block: 000000006bb40000 - 000000006bb40000 (size 0000000000000000) Virtual block: 000000006c4f0000 - 000000006c4f0000 (size 0000000000000000) Virtual block: 000000006dc30000 - 000000006dc30000 (size 0000000000000000) Virtual block: 000000006de10000 - 000000006de10000 (size 0000000000000000) Virtual block: 000000006ef80000 - 000000006ef80000 (size 0000000000000000) Virtual block: 00000000728f0000 - 00000000728f0000 (size 0000000000000000) Virtual block: 0000000071270000 - 0000000071270000 (size 0000000000000000) Virtual block: 0000000074030000 - 0000000074030000 (size 0000000000000000) Virtual block: 00000000746c0000 - 00000000746c0000 (size 0000000000000000) Virtual block: 00000000749d0000 - 00000000749d0000 (size 0000000000000000) Virtual block: 0000000074e50000 - 0000000074e50000 (size 0000000000000000) Virtual block: 0000000071730000 - 0000000071730000 (size 0000000000000000) Virtual block: 0000000075c30000 - 0000000075c30000 (size 0000000000000000) Virtual block: 0000000075e00000 - 0000000075e00000 (size 0000000000000000) Virtual block: 0000000075fe0000 - 0000000075fe0000 (size 0000000000000000) Virtual block: 00000000761a0000 - 00000000761a0000 (size 0000000000000000) Virtual block: 0000000076460000 - 0000000076460000 (size 0000000000000000) Virtual block: 0000000076620000 - 0000000076620000 (size 0000000000000000) Virtual block: 0000000077850000 - 0000000077850000 (size 0000000000000000) Virtual block: 0000000076c60000 - 0000000076c60000 (size 0000000000000000) Virtual block: 0000000078ec0000 - 0000000078ec0000 (size 0000000000000000) Virtual block: 0000000062b30000 - 0000000062b30000 (size 0000000000000000) Virtual block: 000000007a520000 - 000000007a520000 (size 0000000000000000) Virtual block: 000000007ab50000 - 000000007ab50000 (size 0000000000000000) Virtual block: 00000000770e0000 - 00000000770e0000 (size 0000000000000000) Virtual block: 000000007af00000 - 000000007af00000 (size 0000000000000000) Virtual block: 000000007c6b0000 - 000000007c6b0000 (size 0000000000000000) Virtual block: 000000007de50000 - 000000007de50000 (size 0000000000000000) Virtual block: 000000007e2d0000 - 000000007e2d0000 (size 0000000000000000) Virtual block: 000000007fff0000 - 000000007fff0000 (size 0000000000000000) Virtual block: 00000000628c0000 - 00000000628c0000 (size 0000000000000000) Virtual block: 00000000809b0000 - 00000000809b0000 (size 0000000000000000) Virtual block: 0000000080d80000 - 0000000080d80000 (size 0000000000000000) Virtual block: 0000000081310000 - 0000000081310000 (size 0000000000000000) Virtual block: 00000000631a0000 - 00000000631a0000 (size 0000000000000000) Virtual block: 0000000062c70000 - 0000000062c70000 (size 0000000000000000) Virtual block: 0000000092ca0000 - 0000000092ca0000 (size 0000000000000000) Virtual block: 0000000003940000 - 0000000003940000 (size 0000000000000000) Virtual block: 0000000081cd0000 - 0000000081cd0000 (size 0000000000000000) Virtual block: 0000000082260000 - 0000000082260000 (size 0000000000000000) Virtual block: 0000000077200000 - 0000000077200000 (size 0000000000000000) Virtual block: 000000007ec90000 - 000000007ec90000 (size 0000000000000000) Virtual block: 000000007edb0000 - 000000007edb0000 (size 0000000000000000) Virtual block: 000000007eec0000 - 000000007eec0000 (size 0000000000000000) Virtual block: 00000000829f0000 - 00000000829f0000 (size 0000000000000000) Virtual block: 0000000003770000 - 0000000003770000 (size 0000000000000000) Virtual block: 0000000082fc0000 - 0000000082fc0000 (size 0000000000000000) Virtual block: 0000000083210000 - 0000000083210000 (size 0000000000000000) Virtual block: 0000000083320000 - 0000000083320000 (size 0000000000000000) Virtual block: 0000000083760000 - 0000000083760000 (size 0000000000000000) Virtual block: 0000000083870000 - 0000000083870000 (size 0000000000000000) Virtual block: 0000000083eb0000 - 0000000083eb0000 (size 0000000000000000) Virtual block: 0000000085ad0000 - 0000000085ad0000 (size 0000000000000000) Virtual block: 0000000086290000 - 0000000086290000 (size 0000000000000000) 0000000000340000 00000002 210688 174796 210688 60711 775 45 100 1 LFH External fragmentation 34 % (775 free blocks) 0000000000010000 00008000 64 4 64 1 1 1 0 0 00000000005c0000 00001002 3136 1256 3136 278 39 3 0 0 LFH External fragmentation 22 % (39 free blocks) 0000000000960000 00001002 512 8 512 2 1 1 0 0 0000000001520000 00001002 512 276 512 2 10 1 0 0 LFH 00000000013f0000 00001002 512 8 512 2 1 1 0 0 0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH 00000000010b0000 00001002 512 8 512 2 1 1 0 0 00000000019b0000 00001002 512 308 512 5 8 1 0 0 LFH 0000000001220000 00001002 512 380 512 63 21 1 0 0 LFH 00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH 0000000001900000 00001002 512 156 512 133 1 1 0 0 00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH 0000000002e00000 00001002 48256 16656 48256 9836 98 13 0 1 LFH External fragmentation 59 % (98 free blocks) 0000000002f10000 00001002 3584 3280 3584 1773 79 3 0 5 LFH External fragmentation 54 % (79 free blocks) 00000000035c0000 00001002 64 8 64 3 1 1 0 0 0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH 00000000038c0000 00001002 512 8 512 3 1 1 0 0 0000000003ad0000 00001002 512 8 512 3 1 1 0 0 0000000003d40000 00001002 512 8 512 3 1 1 0 0 00000000005b0000 00001002 64 8 64 3 1 1 0 0 0000000004280000 00011002 512 64 512 59 2 1 0 0 00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH 0000000004270000 00001002 1088 304 1088 21 5 2 0 0 LFH 0000000006650000 00000002 1088 80 1088 10 2 2 0 0 0000000007050000 00001002 512 8 512 3 1 1 0 0 0000000007cc0000 00001002 512 8 512 3 1 1 0 0 0000000000be0000 00001002 64 24 64 17 2 1 0 0 0000000006540000 00001002 1536 544 1536 259 4 2 0 0 LFH 0000000000b40000 00001002 64 8 64 3 1 1 0 0 0000000004950000 00001002 512 8 512 3 1 1 0 0 ————————————————————————————-

We see many large virtual blocks allocated with less than 200 KB of normal blocks committed. We suspect possible Memory Leak. Let’s check a few such virtual blocks:

0:000> !address 0000000062b30000

Usage: Heap Base Address: 00000000`62b30000 End Address: 00000000`62c65000 Region Size: 00000000`00135000 ( 1.207 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`62b30000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×62b30000

Content source: 1 (target), length: 135000

0:000> !address 0000000003770000

Usage: Heap Base Address: 00000000`03770000 End Address: 00000000`03870000 Region Size: 00000000`00100000 ( 1.000 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`03770000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×3770000

Content source: 1 (target), length: 100000

0:000> !address 0000000085ad0000

Usage: Heap Base Address: 00000000`85ad0000 End Address: 00000000`8628d000 Region Size: 00000000`007bd000 ( 7.738 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`85ad0000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×85ad0000

Content source: 1 (target), length: 7bd000

We see they all contain assembly packages for KB articles for various updates (Module Hints):

0:000> dc 00000000`85ad0000 00000000`85ad0000 86290000 00000000 83eb0000 00000000 ..)…………. 00000000`85ad0010 00000000 00000000 00000000 00000000 ……………. 00000000`85ad0020 007bd000 00000000 007bd000 00000000 ..{…….{….. 00000000`85ad0030 00000000 00000000 f333297f 04000000 ………)3….. 00000000`85ad0040 00000001 007bc51a 00000004 00000000 ……{……… 00000000`85ad0050 0061003c 00730073 006d0065 006c0062 <.a.s.s.e.m.b.l. 00000000`85ad0060 00200079 006d0078 006e006c 003d0073 y. .x.m.l.n.s.=. 00000000`85ad0070 00750022 006e0072 0073003a 00680063 “.u.r.n.:.s.c.h.

0:000> du 00000000`85ad0050 00000000`85ad0050 “<assembly xmlns=”urn:schemas-mic” 00000000`85ad0090 “rosoft-com:asm.v3″ manifestVersi” 00000000`85ad00d0 “on=”1.0″ description=”Fix for KB” 00000000`85ad0110 “2798162″ displayName=”default” c” 00000000`85ad0150 “ompany=”Microsoft Corporation” c” 00000000`85ad0190 “opyright=”Microsoft Corporation”" 00000000`85ad01d0 ” supportInformation=”http://supp” 00000000`85ad0210 “ort.microsoft.com/?kbid=2798162″” 00000000`85ad0250 ” creationTimeStamp=”2013-04-15T0″ 00000000`85ad0290 “4:10:39Z” lastUpdateTimeStamp=”2″ 00000000`85ad02d0 “013-04-15T04:10:39Z”><assemblyid” 00000000`85ad0310 “entity name=”Package_for_KB27981″

However, in less than 20 minutes CPU and memory consumption normalized and the new saved process memory dump was less than 150 KB. We open it and see that CPU consumption happened for about 15 minutes (in the past) with the consuming thread now #32 instead of #38:

0:000> !runaway f User Mode Time Thread Time 32:14a0 0 days 0:13:24.029 33:b6c 0 days 0:00:00.078 41:174c 0 days 0:00:00.062 40:1bd8 0 days 0:00:00.062 30:fa8 0 days 0:00:00.046 45:650 0 days 0:00:00.015 42:4ac 0 days 0:00:00.015 39:990 0 days 0:00:00.015 29:e0c 0 days 0:00:00.015 28:ff8 0 days 0:00:00.015 18:1304 0 days 0:00:00.015 17:f68 0 days 0:00:00.015 7:664 0 days 0:00:00.015 6:660 0 days 0:00:00.015 4:518 0 days 0:00:00.015 3:4a4 0 days 0:00:00.015 47:9c8 0 days 0:00:00.000 46:f20 0 days 0:00:00.000 44:1440 0 days 0:00:00.000 43:11d0 0 days 0:00:00.000 38:db4 0 days 0:00:00.000 37:6ac 0 days 0:00:00.000 36:c4c 0 days 0:00:00.000 35:ff4 0 days 0:00:00.000 34:1950 0 days 0:00:00.000 31:1910 0 days 0:00:00.000 27:14b8 0 days 0:00:00.000 26:14b4 0 days 0:00:00.000 25:14ac 0 days 0:00:00.000 24:13d0 0 days 0:00:00.000 23:13c8 0 days 0:00:00.000 22:13b4 0 days 0:00:00.000 21:13b0 0 days 0:00:00.000 20:13a8 0 days 0:00:00.000 19:1328 0 days 0:00:00.000 16:e20 0 days 0:00:00.000 15:e10 0 days 0:00:00.000 14:dd0 0 days 0:00:00.000 13:ce0 0 days 0:00:00.000 12:754 0 days 0:00:00.000 11:718 0 days 0:00:00.000 10:678 0 days 0:00:00.000 9:674 0 days 0:00:00.000 8:668 0 days 0:00:00.000 5:548 0 days 0:00:00.000 2:4a0 0 days 0:00:00.000 1:418 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Kernel Mode Time Thread Time 32:14a0 0 days 0:02:24.878 33:b6c 0 days 0:00:00.171 30:fa8 0 days 0:00:00.109 40:1bd8 0 days 0:00:00.093 43:11d0 0 days 0:00:00.078 42:4ac 0 days 0:00:00.062 34:1950 0 days 0:00:00.062 7:664 0 days 0:00:00.062 41:174c 0 days 0:00:00.046 28:ff8 0 days 0:00:00.031 18:1304 0 days 0:00:00.031 16:e20 0 days 0:00:00.031 1:418 0 days 0:00:00.031 45:650 0 days 0:00:00.015 17:f68 0 days 0:00:00.015 6:660 0 days 0:00:00.015 4:518 0 days 0:00:00.015 2:4a0 0 days 0:00:00.015 47:9c8 0 days 0:00:00.000 46:f20 0 days 0:00:00.000 44:1440 0 days 0:00:00.000 39:990 0 days 0:00:00.000 38:db4 0 days 0:00:00.000 37:6ac 0 days 0:00:00.000 36:c4c 0 days 0:00:00.000 35:ff4 0 days 0:00:00.000 31:1910 0 days 0:00:00.000 29:e0c 0 days 0:00:00.000 27:14b8 0 days 0:00:00.000 26:14b4 0 days 0:00:00.000 25:14ac 0 days 0:00:00.000 24:13d0 0 days 0:00:00.000 23:13c8 0 days 0:00:00.000 22:13b4 0 days 0:00:00.000 21:13b0 0 days 0:00:00.000 20:13a8 0 days 0:00:00.000 19:1328 0 days 0:00:00.000 15:e10 0 days 0:00:00.000 14:dd0 0 days 0:00:00.000 13:ce0 0 days 0:00:00.000 12:754 0 days 0:00:00.000 11:718 0 days 0:00:00.000 10:678 0 days 0:00:00.000 9:674 0 days 0:00:00.000 8:668 0 days 0:00:00.000 5:548 0 days 0:00:00.000 3:4a4 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Elapsed Time Thread Time 0:410 0 days 0:35:17.550 1:418 0 days 0:35:17.534 3:4a4 0 days 0:35:17.331 2:4a0 0 days 0:35:17.331 4:518 0 days 0:35:16.957 5:548 0 days 0:35:16.817 6:660 0 days 0:35:15.176 7:664 0 days 0:35:15.156 8:668 0 days 0:35:15.126 9:674 0 days 0:35:15.026 10:678 0 days 0:35:14.986 11:718 0 days 0:35:13.066 12:754 0 days 0:35:13.056 13:ce0 0 days 0:35:08.131 14:dd0 0 days 0:35:07.641 15:e10 0 days 0:35:07.551 16:e20 0 days 0:35:07.531 17:f68 0 days 0:35:06.611 18:1304 0 days 0:35:03.553 19:1328 0 days 0:35:03.381 20:13a8 0 days 0:35:02.024 22:13b4 0 days 0:35:01.931 21:13b0 0 days 0:35:01.931 24:13d0 0 days 0:35:01.899 23:13c8 0 days 0:35:01.899 27:14b8 0 days 0:35:00.932 26:14b4 0 days 0:35:00.932 25:14ac 0 days 0:35:00.932 28:ff8 0 days 0:33:04.785 29:e0c 0 days 0:33:04.644 30:fa8 0 days 0:32:59.750 31:1910 0 days 0:32:55.608 32:14a0 0 days 0:32:12.276 34:1950 0 days 0:18:39.607 33:b6c 0 days 0:18:39.607 35:ff4 0 days 0:17:43.530 36:c4c 0 days 0:08:45.458 38:db4 0 days 0:07:41.551 37:6ac 0 days 0:07:41.551 39:990 0 days 0:06:54.877 40:1bd8 0 days 0:06:54.867 41:174c 0 days 0:05:38.282 42:4ac 0 days 0:03:54.627 43:11d0 0 days 0:03:53.122 44:1440 0 days 0:03:51.627 45:650 0 days 0:02:15.536 47:9c8 0 days 0:00:11.100 46:f20 0 days 0:00:11.100

0:000> ~32k # Child-SP RetAddr Call Site 00 00000000`0414f558 000007fe`fd4f1430 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`0414f560 00000000`775706e0 KERNELBASE!WaitForMultipleObjectsEx+0xe8 *** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll - 02 00000000`0414f660 000007fe`ee7de250 kernel32!WaitForMultipleObjects+0xb0 03 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0x4e80 04 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd 05 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Process heap doesn’t have any large allocated virtual block with only almost 100 KB committed normal blocks:

0:000> !heap -s

**** NT HEAP STATS BELOW ****************************** LFH Key Termination on corruption Heap Flags (k) (k) (k) (k) length ------------------------- 0000000000340000 External fragmentation 0000000000010000 00008000 00000000005c0000 00001002 External fragmentation 0000000000960000 00001002 512 0000000001520000 00001002 00000000013f0000 00001002 512 0000000001790000 00001002 00000000010b0000 00001002 512 00000000019b0000 00001002 0000000001220000 00001002 00000000021e0000 00001002 0000000001900000 00001002 00000000024b0000 00001002 0000000002e00000 00001002 External fragmentation 0000000002f10000 00001002 External fragmentation 00000000035c0000 00001002 0000000003200000 00001002 00000000038c0000 00001002 512 0000000003ad0000 00001002 512 0000000003d40000 00001002 512 00000000005b0000 00001002 0000000004280000 00011002 00000000040c0000 00001002 0000000004270000 00001002 0000000007050000 00001002 512 0000000007cc0000 00001002 512 0000000000be0000 00001002 Virtual block: 0000000009b90000 0000000006540000 00001002 External fragmentation 0000000000b40000 00001002 0000000004950000 00001002 ———R ************************** : 0x000000a57ddeb5ed : ENABLED Reserv Commit Virt Free List UCR Virt Lock Fast blocks cont. heap ------------------------------------------------------------ 00000002 210688 94444 210688 85646 752 86 0 1 LFH 90 % (752 free blocks) 64 4 64 1 1 1 0 0 3136 1256 3136 282 39 3 0 0 LFH 22 % (39 free blocks) 8 512 2 1 1 0 0 512 280 512 3 10 1 0 0 LFH 8 512 2 1 1 0 0 512 248 512 0 6 1 0 0 LFH 8 512 2 1 1 0 0 512 320 512 6 12 1 0 0 LFH 512 380 512 63 22 1 0 0 LFH 512 256 512 11 3 1 0 0 LFH 512 156 512 133 1 1 0 0 1536 792 1536 27 4 2 0 0 LFH 48256 16656 48256 9870 101 13 0 1 LFH 59 % (101 free blocks) 3584 3280 3584 1805 73 3 0 5 LFH 55 % (73 free blocks) 64 8 64 3 1 1 0 0 512 388 512 2 3 1 0 0 LFH 8 512 3 1 1 0 0 8 512 3 1 1 0 0 8 512 3 1 1 0 0 64 8 64 3 1 1 0 0 512 64 512 59 2 1 0 0 1088 296 1088 14 6 2 0 0 LFH 1088 304 1088 22 6 2 0 0 LFH 8 512 3 1 1 0 0 8 512 3 1 1 0 0 64 24 64 17 2 1 0 0 - 0000000009b90000 (size 0000000000000000) 1536 1064 1536 775 11 2 1 0 LFH 72 % (11 free blocks) 64 8 64 3 1 1 0 0 512 312 512 55 12 1 0 0 LFH 12;————————————————————————-

So we consider the dump set as an instance of Memory Fluctuation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 27e)

Monday, August 24th, 2015

This is another variant of Stack Trace Collection pattern that shows stack traces from threads currently execution on all CPUs. Although we can see the non-idle running threads from the stack traces corresponding to all processes and their threads we may also want to see idle thread stack traces too. Also, the corresponding WinDbg command (!running -t -i) is faster if we want to double check the output of !analyze -v command in case of BSOD. The latter command may show the stack trace from the current CPU instead of the stack trace from the thread running on a different CPU that caused a bugcheck. Here’s an example from one of the memory dumps for which !analyze -v command shows an incorrect stack trace in the output when we open the dump file. It reports the stack trace from CPU 0 but the bugcheck happened on CPU 1:

0: kd> !running -t -i

System Processors: (00000000000000ff) Idle Processors: (00000000000000fd)

Prcbs Current (pri) Next (pri) Idle 0 fffff801e5d85180 fffff801e5ddea00 ( 0) fffff801e5ddea00 ................

Child-SP RetAddr Call Site fffff801`e8c9eb60 fffff801`e5c69b74 hal!KeQueryPerformanceCounter+0x75 fffff801`e8c9eba0 fffff801`e5c69e01 nt!KiCheckStall+0x2c fffff801`e8c9ebd0 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 fffff801`e8c9ece0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b fffff801`e8c9ed30 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 fffff801`e8c9ee70 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 fffff801`e8c8c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf fffff801`e8c8c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa fffff801`e8c8c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 fffff801`e8c8cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 fffff801`e8c8cc60 00000000`00000000 nt!KiIdleLoop+0x2c

1 ffffd000f0975180 ffffe0000d726880 (12) ffffd000f09813c0 …………….

Child-SP RetAddr Call Site ffffd000`202cb618 fffff801`e5a1cc3c hal!HalpAcpiPmRegisterReadPort+0x1b ffffd000`202cb620 fffff801`e5a417e7 hal!HalpAcpiPmRegisterRead+0x30 ffffd000`202cb650 fffff801`e5c66af5 hal!HaliHaltSystem+0x53 ffffd000`202cb690 fffff801`e5c66741 nt!KiBugCheckDebugBreak+0×99 ffffd000`202cb6f0 fffff801`e5bd2aa4 nt!KeBugCheck2+0xc6d ffffd000`202cbe00 fffff801`e5bde4e9 nt!KeBugCheckEx+0×104 ffffd000`202cbe40 fffff801`e5bdcd3a nt!KiBugCheckDispatch+0×69 ffffd000`202cbf80 fffff800`913601da nt!KiPageFault+0×23a ffffd000`202cc118 fffff800`91363710 DriverA!memcpy+0×21a […]

2 ffffd000f09ee180 ffffd000f09fa3c0 ( 0) ffffd000f09fa3c0 ................

Child-SP RetAddr Call Site ffffd000`f09f9f88 fffff801`e5c69e01 nt!KiCheckStall+0xa ffffd000`f09f9f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`f09fa0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`f09fa0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`f09fa230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb5938e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb5938f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb593920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb593b10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb593c60 00000000`00000000 nt!KiIdleLoop+0x2c

3 ffffd000eb5e5180 ffffd000eb5f13c0 ( 0) ffffd000eb5f13c0 ................

Child-SP RetAddr Call Site ffffd000`eb5f0f60 fffff801`e5c69e01 nt!KiCheckStall+0x5f ffffd000`eb5f0f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb5f10a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb5f10f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb5f1230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb5fa8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb5fa8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb5fa920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb5fab10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb5fac60 00000000`00000000 nt!KiIdleLoop+0x2c

4 ffffd000f08d1180 ffffd000f08dd3c0 ( 0) ffffd000f08dd3c0 ................

Child-SP RetAddr Call Site ffffd000`f08dcf90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x227 ffffd000`f08dd0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`f08dd0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`f08dd230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb85b8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb85b8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb85b920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb85bb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb85bc60 00000000`00000000 nt!KiIdleLoop+0x2c

5 ffffd000eb8ad180 ffffd000eb8b93c0 ( 0) ffffd000eb8b93c0 ................

Child-SP RetAddr Call Site ffffd000`eb8b8f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb8b8f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb8b90a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb8b90f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb8b9230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb8db8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb8db8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb8db920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb8dbb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb8dbc60 00000000`00000000 nt!KiIdleLoop+0x2c

6 ffffd000eb92a180 ffffd000eb9363c0 ( 0) ffffd000eb9363c0 ................

Child-SP RetAddr Call Site ffffd000`eb935f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb935f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb9360a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb9360f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb936230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb93f8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb93f8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb93f920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb93fb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb93fc60 00000000`00000000 nt!KiIdleLoop+0x2c

7 ffffd000eb967180 ffffd000eb9733c0 ( 0) ffffd000eb9733c0 ................

Child-SP RetAddr Call Site ffffd000`eb972f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb972f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb9730a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb9730f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb973230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb97c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb97c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb97c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb97cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb97cc60 00000000`00000000 nt!KiIdleLoop+0x2c

This command is obviously faster than repeatedly switching to subsequent CPUs using ~s command and then checking the corresponding stack trace (k). It also helps in diagnosing Spiking Threads in kernel and complete memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Kernel Memory Dump Analysis, x64 Windows | 1 Comment »

Crash Dump Analysis Patterns (Part 224)

Saturday, April 25th, 2015

When analyzing Spiking Threads across Snapshot Collection we are interested in finding a module (or a function) that was most likely responsible (for example, “looping” inside). Here we can compare the same thread stack trace from different memory dumps and find their Variable Subtrace. For such subtraces we have changes in kv-style output: in return addresses, stack frame values, and possible arguments. The call site that starts the variable subtrace is the most likely candidate (subject to the number of snapshots). For example, consider the following pseudo code:

ModuleA!start()
{
    ModuleA!func1();
}

ModuleA!func1()
{
    ModuleB!func2();
}

ModuleB!func2()
{
    while (…)
    {
        ModuleB!func3();
    }
}

ModuleB!func3()
{
    ModuleB!func4();
}

ModuleB!func4()
{
    ModuleB!func5();
}

ModuleB!func5()
{
    // ...
}

Here, the variable stack trace part will correspond to ModuleB frames. The memory dump can be saved anywhere inside the “while” loop and down the calls, and the last variable return address down the stack trace will belong to ModuleB!func2 address range. The non-variable part will start with ModuleA!func1 address range:

// snapshot 1

RetAddr ModuleB!func4+0×20 ModuleB!func3+0×10 ModuleB!func2+0×40 ModuleA!func1+0×10 ModuleA!start+0×300

// snapshot 2

RetAddr ModuleB!func2+0×20 ModuleA!func1+0×10 ModuleA!start+0×300

// snapshot 3

RetAddr ModuleB!func3+0×20 ModuleB!func2+0×40 ModuleA!func1+0×10 ModuleA!start+0×300

To illustrate this analysis pattern we adopted Memory Cell Diagram (MCD) approach from Accelerated Disassembly, Reconstruction and Reversing training and introduce here Abstract Stack Trace Notation (ASTN) diagrams where different colors are used for different modules and changes are highlighted with different fill patterns. The following three ASTN diagrams from subsequently saved process memory dumps illustrate real stack traces we analyzed some time ago. We see that the variable subtrace contains only the 3rd-party ModuleB calls. Moreover, the loop is possibly contained inside ModuleB because all ModuleA frames are non-variable including Child-SP and Args column values.

If we had ASTN diagrams below instead we would have concluded that the loop was in ModuleA with changes in ModuleB columns as an execution side effect:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Abstract Stack Trace Notation, Crash Dump Analysis, Crash Dump Patterns, Memory Cell Diagrams, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 223)

Saturday, March 7th, 2015

Sometimes we can see signs of Crashed Processes in kernel and complete memory dumps. By crashes we mean the sudden disappearance of processes from Task Manager, for example. In memory dumps we can still see such processes as Zombie Processes. Special Processes found in the process list may help to select the possible candidate among many Zombie Processes. If a process is supposed to be launched only once (like a service) but found several times as Zombie Process and also as a normal process later in the process list (for example, as Last Object), then this may point to possible past crashes (or silent terminations). We also have a similar trace analysis pattern: Singleton Event. The following example illustrates both signs:

0: kd> !process 0 0

[...]

PROCESS fffffa80088a5640 SessionId: 0 Cid: 2184 Peb: 7fffffd7000 ParentCid: 0888 DirBase: 381b8000 ObjectTable: 00000000 HandleCount: 0. Image: WerFault.exe

PROCESS fffffa8007254b30 SessionId: 0 Cid: 20ac Peb: 7fffffdf000 ParentCid: 02cc DirBase: b3306000 ObjectTable: 00000000 HandleCount: 0. Image: ServiceA.exe

[...]

PROCESS fffffa8007fe2b30 SessionId: 0 Cid: 2a1c Peb: 7fffffdf000 ParentCid: 02cc DirBase: 11b649000 ObjectTable: fffff8a014939530 HandleCount: 112. Image: ServiceA.exe

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 221)

Saturday, February 21st, 2015

Corrupt Structure is added for completeness of pattern discourse. We mentioned it a few times, for example, in Self-Diagnosis (kernel mode), and Critical Section Corruption. Typical signs of the corrupt structure include:

- Regular Data such as ASCII and UNICODE fragments over substructures and pointer areas
- Large values where you expect small and vice versa
- User space address values where we expect kernel space and vice versa
- Malformed and partially zeroed _LIST_ENTRY data (see exercise C3 for linked list navigation)
- Memory read errors for pointer dereferences or inaccessible memory indicators (??)
- Memory read error at the end of the linked list while traversing structures

0: kd> dt _ERESOURCE ffffd0002299f830 ntdll!_ERESOURCE +0x000 SystemResourcesList : _LIST_ENTRY [ 0xffffc000`07b64800 - 0xffffe000`02a79970 ] +0x010 OwnerTable : 0xffffe000`02a79940 _OWNER_ENTRY +0x018 ActiveCount : 0n0 +0x01a Flag : 0 +0x01a ReservedLowFlags : 0 '' +0x01b WaiterPriority : 0 '' +0x020 SharedWaiters : 0x00000000`00000001 _KSEMAPHORE +0x028 ExclusiveWaiters : 0xffffe000`02a79a58 _KEVENT +0x030 OwnerEntry : _OWNER_ENTRY +0x040 ActiveEntries : 0 +0x044 ContentionCount : 0 +0×048 NumberOfSharedWaiters : 0×7b64800 +0×04c NumberOfExclusiveWaiters : 0xffffc000 +0×050 Reserved2 : (null) +0×058 Address : 0xffffd000`2299f870 Void +0×058 CreatorBackTraceIndex : 0xffffd000`2299f870 +0×060 SpinLock : 1

0: kd> dt _ERESOURCE ffffd0002299d830 ntdll!_ERESOURCE +0×000 SystemResourcesList : _LIST_ENTRY [ 0×000001e0`00000280 - 0×00000000`00000004 ] +0×010 OwnerTable : 0×00000000`0000003c _OWNER_ENTRY +0×018 ActiveCount : 0n0 +0×01a Flag : 0 +0×01a ReservedLowFlags : 0 ” +0×01b WaiterPriority : 0 ” +0×020 SharedWaiters : 0×0000003c`000001e0 _KSEMAPHORE +0×028 ExclusiveWaiters : (null) +0×030 OwnerEntry : _OWNER_ENTRY +0×040 ActiveEntries : 0 +0×044 ContentionCount : 0×7f +0×048 NumberOfSharedWaiters : 0×7f +0×04c NumberOfExclusiveWaiters : 0×7f +0×050 Reserved2 : 0×00000001`00000001 Void +0×058 Address : 0×00000000`00000005 Void +0×058 CreatorBackTraceIndex : 5 +0×060 SpinLock : 0

However, we need to be sure that we supplied the correct pointer to dt WinDbg command. One of the signs that the pointer was incorrect are memory read errors or all zeroes:

0: kd> dt _ERESOURCE ffffd000229af830 ntdll!_ERESOURCE +0x000 SystemResourcesList : _LIST_ENTRY [ 0x00000000`00000000 - 0x00000000`00000000 ] +0x010 OwnerTable : (null) +0x018 ActiveCount : 0n0 +0x01a Flag : 0 +0x01a ReservedLowFlags : 0 '' +0x01b WaiterPriority : 0 '' +0x020 SharedWaiters : (null) +0x028 ExclusiveWaiters : (null) +0x030 OwnerEntry : _OWNER_ENTRY +0x040 ActiveEntries : 0 +0x044 ContentionCount : 0 +0x048 NumberOfSharedWaiters : 0 +0x04c NumberOfExclusiveWaiters : 0 +0x050 Reserved2 : (null) +0x058 Address : (null) +0x058 CreatorBackTraceIndex : 0 +0x060 SpinLock : 0

0: kd> dt _ERESOURCE ffffd00022faf830 ntdll!_ERESOURCE +0x000 SystemResourcesList : _LIST_ENTRY +0x010 OwnerTable : ???? +0x018 ActiveCount : ?? +0x01a Flag : ?? +0x01a ReservedLowFlags : ?? +0x01b WaiterPriority : ?? +0x020 SharedWaiters : ???? +0x028 ExclusiveWaiters : ???? +0x030 OwnerEntry : _OWNER_ENTRY +0x040 ActiveEntries : ?? +0x044 ContentionCount : ?? +0x048 NumberOfSharedWaiters : ?? +0x04c NumberOfExclusiveWaiters : ?? +0x050 Reserved2 : ???? +0x058 Address : ???? +0x058 CreatorBackTraceIndex : ?? +0x060 SpinLock : ?? Memory read error ffffd00022faf890

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 218)

Wednesday, December 31st, 2014

Objects such as processes may be referenced internally in addition to using handles. If their reference counts are unbalanced we may have Reference Leak pattern. For example, we have an instance of thousands of Zombie Processes but we don’t see Handle Leaks from their parent processes if we analyze ParentCids:

0: kd> !process 0 0 [...] PROCESS fffffa801009a060 SessionId: 0 Cid: 2e270 Peb: 7fffffdb000 ParentCid: 032c DirBase: 12ba37000 ObjectTable: 00000000 HandleCount: 0. Image: conhost.exe

PROCESS fffffa8009b7e8e0 SessionId: 1 Cid: 2e0c8 Peb: 7fffffd9000 ParentCid: 10a0 DirBase: 21653e000 ObjectTable: 00000000 HandleCount: 0. Image: taskmgr.exe

PROCESS fffffa8009e7a450 SessionId: 0 Cid: 2e088 Peb: 7efdf000 ParentCid: 0478 DirBase: 107f02000 ObjectTable: 00000000 HandleCount: 0. Image: AppA.exe

PROCESS fffffa8009e794b0 SessionId: 0 Cid: 2e394 Peb: 7fffffd3000 ParentCid: 032c DirBase: 210ffc000 ObjectTable: 00000000 HandleCount: 0. Image: conhost.exe

PROCESS fffffa8009ed4060 SessionId: 0 Cid: 2dee4 Peb: 7efdf000 ParentCid: 0478 DirBase: 11b7c7000 ObjectTable: 00000000 HandleCount: 0. Image: AppB.exe

PROCESS fffffa800a13bb30 SessionId: 0 Cid: 2e068 Peb: 7fffffd5000 ParentCid: 032c DirBase: 1bb8c1000 ObjectTable: 00000000 HandleCount: 0. Image: conhost.exe

PROCESS fffffa80096f26b0 SessionId: 0 Cid: 2e320 Peb: 7efdf000 ParentCid: 0478 DirBase: 6ad4c000 ObjectTable: 00000000 HandleCount: 0. Image: AppC.exe

PROCESS fffffa8009c44060 SessionId: 0 Cid: 2e300 Peb: 7fffffdd000 ParentCid: 032c DirBase: 10df06000 ObjectTable: 00000000 HandleCount: 0. Image: conhost.exe [...]

0: kd> !object fffffa800a13bb30 Object: fffffa800a13bb30 Type: (fffffa8006cecf30) Process ObjectHeader: fffffa800a13bb00 (new version) HandleCount: 0 PointerCount: 1

0: kd> !object fffffa8009b7e8e0 Object: fffffa8009b7e8e0 Type: (fffffa8006cecf30) Process ObjectHeader: fffffa8009b7e8b0 (new version) HandleCount: 0 PointerCount: 1

Such number of processes correlates with non-paged pool usage for process structures:

0: kd> !poolused 3 .... Sorting by NonPaged Pool Consumed

NonPaged Paged Tag Allocs Frees Diff Used Allocs Frees Diff Used

Proc 55488 60 55428 80328320 0 0 0 0 Process objects , Binary: nt!ps File 51733526 51708737 24789 7150416 0 0 0 0 File objects […]

Here we recommend enabling object reference tracing either using gflags.exe or directly modifying registry:

Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel Value: ObTracePoolTags Type: REG_SZ Data: Proc

Note: after troubleshooting or debugging please disable tracing because it consumes pool (another variant of Instrumentation Side Effect pattern and may lead to similar Insufficient Memory pattern for stack trace database):

0: kd> !poolused 3 .... Sorting by NonPaged Pool Consumed

NonPaged Paged Tag Allocs Frees Diff Used Allocs Frees Diff Used

ObRt 5688634 5676109 12525 4817288240 0 0 0 0 object reference stack tracing , Binary: nt!ob Proc 22120 101 22019 25961168 0 0 0 0 Process objects , Binary: nt!ps […]

After enabling tracing we collect a complete memory dump (in case of postmortem debugging) to analyze another variant of Stack Trace pattern using !obtrace WinDbg command:

0: kd> !obtrace fffffa800af9e220 Object: fffffa800af9e220 Image: AppD.exe Sequence (+/-) Tag Stack -------- ----- ---- --------------------------------------------------- ad377858 +1 Dflt nt! ?? ::NNGAKEGL::`string'+21577 nt!PspAllocateProcess+185 nt!NtCreateUserProcess+4a3 nt!KiSystemServiceCopyEnd+13

ad37787d +1 Dflt nt! ?? ::FNODOBFM::`string'+18f1d nt!NtCreateUserProcess+569 nt!KiSystemServiceCopyEnd+13

ad377882 +1 Dflt nt! ?? ::NNGAKEGL::`string'+1f9d8 nt!NtProtectVirtualMemory+119 nt!KiSystemServiceCopyEnd+13 nt!KiServiceLinkage+0 nt!RtlCreateUserStack+1e4 nt!PspAllocateThread+299 nt!NtCreateUserProcess+65d nt!KiSystemServiceCopyEnd+13

ad377884 -1 Dflt nt! ?? ::FNODOBFM::`string'+4886e nt!NtProtectVirtualMemory+161 nt!KiSystemServiceCopyEnd+13 nt!KiServiceLinkage+0 nt!RtlCreateUserStack+1e4 [...]

Analysis of such traces may be complicated due to Truncated Stack Traces. We plan to show one counting trick in the next pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 216)

Wednesday, November 19th, 2014

Sometimes what looks like a memory leak when we install the new product version is not really a leak. With the previous version we had 400 MB typical memory usage but suddenly we get twice as more. We shouldn’t panic but collect a process memory dump to calmly inspect it offline. We may see Dry Weight increase: the size of all module images. For some products the new release may mean complete redesign with a new more powerful framework or incorporation of the significant number of new 3rd-party components (Module Variety). Additional sign against the memory leak hypothesis is simultaneous memory usage increase for many product processes. Although, this may be some shared module with leaking code. For example, in the example below 50% of all committed memory was image memory:

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal [...] Image 1806 0`19031000 ( 402.535 Mb) 4.29% 0.00% Heap 72 0`02865000 ( 40.395 Mb) 0.44% 0.00% […]

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal [...] MEM_IMAGE 2281 0`19AA8000 ( 413.000 Mb) 4.40% 0.00% […]

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal [...] MEM_COMMIT 2477 0`326e8000 ( 806.906 Mb) 8.76% 0.00% […]

WinDbg lmt command shows almost 50 new .NET components.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 215)

Sunday, November 16th, 2014

Sometimes Wait Chains such as involving critical sections may have a Missing Thread endpoint. But in some cases we might see a Ghost Thread whose TID was reused by subsequent thread creation in a different process. For example, critical section structure may refer to such TID as in the example below.

// Critical section from LSASS process

THREAD fffffa803431cb50 Cid 03e8.2718 Teb: 000007fffff80000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable fffffa80330e0500 SynchronizationEvent Impersonation token: fffff8a00b807060 (Level Impersonation) Owning Process fffffa8032354c40 Image: lsass.exe Attached Process N/A Image: N/A Wait Start TickCount 107175 Ticks: 19677 (0:00:05:06.963) Context Switch Count 2303 IdealProcessor: 1 UserTime 00:00:00.218 KernelTime 00:00:00.109 Win32 Start Address ntdll!TppWorkerThread (0×0000000076e1f2e0) Stack Init fffff88008e5fdb0 Current fffff88008e5f900 Base fffff88008e60000 Limit fffff88008e5a000 Call 0 Priority 10 BasePriority 10 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Kernel stack not resident. Child-SP RetAddr Call Site fffff880`08e5f940 fffff800`01c7cf72 nt!KiSwapContext+0×7a fffff880`08e5fa80 fffff800`01c8e39f nt!KiCommitThreadWait+0×1d2 fffff880`08e5fb10 fffff800`01f7fe3e nt!KeWaitForSingleObject+0×19f fffff880`08e5fbb0 fffff800`01c867d3 nt!NtWaitForSingleObject+0xde fffff880`08e5fc20 00000000`76e5067a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`08e5fc20) 00000000`0427cca8 00000000`76e4d808 ntdll!NtWaitForSingleObject+0xa 00000000`0427ccb0 00000000`76e4d6fb ntdll!RtlpWaitOnCriticalSection+0xe8 00000000`0427cd60 000007fe`f46a4afe ntdll!RtlEnterCriticalSection+0xd1 […]

1: kd> .process /r /p fffffa8032354c40 Implicit process is now fffffa80`32353b30 Loading User Symbols

1: kd> !cs -l -o -s ----------------------------------------- DebugInfo = 0x0000000003475220 Critical section = 0x0000000003377740 (+0x3377740) LOCKED LockCount = 0×10 WaiterWoken = No OwningThread = 0×00000000000004e4 RecursionCount = 0×0 LockSemaphore = 0×0 SpinCount = 0×0000000000000000 OwningThread = .thread fffffa80344e4c00 […]

// The "owner" thread is from winlogon.exe

1: kd> !thread fffffa80344e4c00 3f THREAD fffffa80344e4c00 Cid 21d0.14e4 Teb: 000007fffffae000 Win32Thread: fffff900c0998c20 WAIT: (WrUserRequest) UserMode Non-Alertable fffffa80355817d0 SynchronizationEvent Not impersonating DeviceMap fffff8a0000088f0 Owning Process fffffa8034ff77c0 Image: winlogon.exe […]

A PML (Process Monitor) log was recorded before the complete memory dump was forced, and it clearly shows Glued Activity trace analysis pattern. The thread was owned by LSASS but then exited and its TID was subsequently reused by 2 other processes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Software Trace Analysis, Trace Analysis Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 61b)

Saturday, November 15th, 2014

We recently encountered Invalid Handle pattern in the context of .NET program execution. We decided to model it and wrote a small C# program:

namespace SafeHandle
{
    class Program
    {
        static void Main(string[] args)
        {
            SafeFileHandle hFile =
                 new SafeFileHandle(new IntPtr(0xDEAD), true);
            Console.WriteLine("About to close");
            Console.ReadKey();
        }
    }
}

Of course, when we execute it nothing happens. Invalid handles are ignored by default. However, to change the behavior we enabled “Enable close exception” in glags.exe:

And if we run it we get this Managed Stack Trace:

We could have detected invalid handle if we enabled Application Verifier but then we wouldn’t have Managed Code Exception.

So we load a crash dump (saved because we enabled LocalDumps) and load SOS extension:

0:002> lmv m clr start end module name 000007fe`ed880000 000007fe`ee1eb000 clr (pdb symbols) Loaded symbol image file: clr.dll Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll [...]

0:002> .load C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos

0:002> !pe Exception object: 0000000002ab5fe8 Exception type: System.Runtime.InteropServices.SEHException Message: External component has thrown an exception. InnerException: StackTrace (generated): SP IP Function 000000001B40EDD0 0000000000000000 mscorlib_ni!Microsoft.Win32.Win32Native.CloseHandle(IntPtr)+0×1 000000001B40F2F0 0000000000000000 mscorlib_ni!System.Runtime.InteropServices.SafeHandle.InternalFinalize()+0×1 000000001B40F2F0 000007FEEC62F7A6 mscorlib_ni!System.Runtime.InteropServices.SafeHandle.Finalize()+0×26

StackTraceString: HResult: 80004005

Our unmanaged CLR Thread Exception Stack Trace is quite simple:

0:002> k Child-SP RetAddr Call Site 00000000`1b40d6e8 000007fe`fd651430 ntdll!NtWaitForMultipleObjects+0xa 00000000`1b40d6f0 00000000`77621723 KERNELBASE!WaitForMultipleObjectsEx+0xe8 00000000`1b40d7f0 00000000`7769b5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3 00000000`1b40d880 00000000`7769b767 kernel32!WerpReportFaultInternal+0x215 00000000`1b40d920 00000000`7769b7bf kernel32!WerpReportFault+0x77 00000000`1b40d950 00000000`7769b9dc kernel32!BasepReportFault+0x1f 00000000`1b40d980 00000000`778b3398 kernel32!UnhandledExceptionFilter+0x1fc 00000000`1b40da60 00000000`778385c8 ntdll! ?? ::FNODOBFM::`string'+0x2365 00000000`1b40da90 00000000`77849d2d ntdll!_C_specific_handler+0x8c 00000000`1b40db00 00000000`778391cf ntdll!RtlpExecuteHandlerForException+0xd 00000000`1b40db30 00000000`778397c8 ntdll!RtlDispatchException+0x45a 00000000`1b40e210 00000000`778712c7 ntdll!RtlRaiseException+0x22f 00000000`1b40ebc0 000007fe`fd651873 ntdll!KiRaiseUserExceptionDispatcher+0×3a 00000000`1b40ec90 00000000`77621991 KERNELBASE!CloseHandle+0×13 00000000`1b40ecc0 000007fe`ec720418 kernel32!CloseHandleImplementation+0×3d 00000000`1b40edd0 000007fe`ed8e9e03 mscorlib_ni+0×580418 00000000`1b40eea0 000007fe`ed8e9e7e clr!CallDescrWorkerInternal+0×83 00000000`1b40eee0 000007fe`ed8ec860 clr!CallDescrWorkerWithHandler+0×4a 00000000`1b40ef20 000007fe`ed8f1a1d clr!DispatchCallSimple+0×85 00000000`1b40efb0 000007fe`ed8f19ac clr!SafeHandle::RunReleaseMethod+0×69 00000000`1b40f050 000007fe`ed8f180a clr!SafeHandle::Release+0×122 00000000`1b40f120 000007fe`eda4863e clr!SafeHandle::Dispose+0×36 00000000`1b40f190 000007fe`ec62f7a6 clr!SafeHandle::Finalize+0xa2 00000000`1b40f2f0 000007fe`ed8e9d56 mscorlib_ni+0×48f7a6 00000000`1b40f330 000007fe`eda83c4e clr!FastCallFinalizeWorker+0×6 00000000`1b40f360 000007fe`eda83bc3 clr!MethodDesc::RequiresFullSlotNumber+0×72 00000000`1b40f3a0 000007fe`eda83b0f clr!MethodTable::CallFinalizer+0xa3 00000000`1b40f3e0 000007fe`ed9fee46 clr!SVR::CallFinalizer+0×5f 00000000`1b40f420 000007fe`ed9aac5b clr!SVR::CallFinalizer+0×102 00000000`1b40f4e0 000007fe`ed8f458c clr!WKS::GCHeap::IsPromoted+0xee 00000000`1b40f520 000007fe`ed8f451a clr!Frame::Pop+0×50 00000000`1b40f560 000007fe`ed8f4491 clr!COMCustomAttribute::PopSecurityContextFrame+0×192 00000000`1b40f660 000007fe`ed9d1bfe clr!COMCustomAttribute::PopSecurityContextFrame+0xbd 00000000`1b40f6f0 000007fe`ed9d1e59 clr!ManagedThreadBase_NoADTransition+0×3f 00000000`1b40f750 000007fe`ed9533de clr!WKS::GCHeap::FinalizerThreadStart+0×193 00000000`1b40f790 00000000`776159ed clr!Thread::intermediateThreadProc+0×7d 00000000`1b40f850 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd 00000000`1b40f880 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that exception processing happened during object finalization. We can infer the value of the handle (may be Small Value) via disassembly if this is possible:

0:002> kn # Child-SP RetAddr Call Site 00 00000000`1b40d6e8 000007fe`fd651430 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`1b40d6f0 00000000`77621723 KERNELBASE!WaitForMultipleObjectsEx+0xe8 02 00000000`1b40d7f0 00000000`7769b5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3 03 00000000`1b40d880 00000000`7769b767 kernel32!WerpReportFaultInternal+0x215 04 00000000`1b40d920 00000000`7769b7bf kernel32!WerpReportFault+0x77 05 00000000`1b40d950 00000000`7769b9dc kernel32!BasepReportFault+0x1f 06 00000000`1b40d980 00000000`778b3398 kernel32!UnhandledExceptionFilter+0x1fc 07 00000000`1b40da60 00000000`778385c8 ntdll! ?? ::FNODOBFM::`string'+0x2365 08 00000000`1b40da90 00000000`77849d2d ntdll!_C_specific_handler+0x8c 09 00000000`1b40db00 00000000`778391cf ntdll!RtlpExecuteHandlerForException+0xd 0a 00000000`1b40db30 00000000`778397c8 ntdll!RtlDispatchException+0x45a 0b 00000000`1b40e210 00000000`778712c7 ntdll!RtlRaiseException+0x22f 0c 00000000`1b40ebc0 000007fe`fd651873 ntdll!KiRaiseUserExceptionDispatcher+0x3a 0d 00000000`1b40ec90 00000000`77621991 KERNELBASE!CloseHandle+0×13 0e 00000000`1b40ecc0 000007fe`ec720418 kernel32!CloseHandleImplementation+0×3d 0f 00000000`1b40edd0 000007fe`ed8e9e03 mscorlib_ni+0×580418 10 00000000`1b40eea0 000007fe`ed8e9e7e clr!CallDescrWorkerInternal+0×83 11 00000000`1b40eee0 000007fe`ed8ec860 clr!CallDescrWorkerWithHandler+0×4a 12 00000000`1b40ef20 000007fe`ed8f1a1d clr!DispatchCallSimple+0×85 13 00000000`1b40efb0 000007fe`ed8f19ac clr!SafeHandle::RunReleaseMethod+0×69 14 00000000`1b40f050 000007fe`ed8f180a clr!SafeHandle::Release+0×122 15 00000000`1b40f120 000007fe`eda4863e clr!SafeHandle::Dispose+0×36 16 00000000`1b40f190 000007fe`ec62f7a6 clr!SafeHandle::Finalize+0xa2 17 00000000`1b40f2f0 000007fe`ed8e9d56 mscorlib_ni+0×48f7a6 18 00000000`1b40f330 000007fe`eda83c4e clr!FastCallFinalizeWorker+0×6 19 00000000`1b40f360 000007fe`eda83bc3 clr!MethodDesc::RequiresFullSlotNumber+0×72 1a 00000000`1b40f3a0 000007fe`eda83b0f clr!MethodTable::CallFinalizer+0xa3 1b 00000000`1b40f3e0 000007fe`ed9fee46 clr!SVR::CallFinalizer+0×5f 1c 00000000`1b40f420 000007fe`ed9aac5b clr!SVR::CallFinalizer+0×102 1d 00000000`1b40f4e0 000007fe`ed8f458c clr!WKS::GCHeap::IsPromoted+0xee 1e 00000000`1b40f520 000007fe`ed8f451a clr!Frame::Pop+0×50 1f 00000000`1b40f560 000007fe`ed8f4491 clr!COMCustomAttribute::PopSecurityContextFrame+0×192 20 00000000`1b40f660 000007fe`ed9d1bfe clr!COMCustomAttribute::PopSecurityContextFrame+0xbd 21 00000000`1b40f6f0 000007fe`ed9d1e59 clr!ManagedThreadBase_NoADTransition+0×3f 22 00000000`1b40f750 000007fe`ed9533de clr!WKS::GCHeap::FinalizerThreadStart+0×193 23 00000000`1b40f790 00000000`776159ed clr!Thread::intermediateThreadProc+0×7d 24 00000000`1b40f850 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd 25 00000000`1b40f880 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:002> .frame /c d 0d 00000000`1b40ec90 00000000`77621991 KERNELBASE!CloseHandle+0x13 rax=00000000c0000001 rbx=000000000000dead rcx=00000000009a0000 rdx=0000000000000001 rsi=000000001b40efd0 rdi=000000001b40eff8 rip=000007fefd651873 rsp=000000001b40ec90 rbp=000000001b40edf0 r8=000000001b40ce08 r9=000000001b40cf70 r10=0000000000000000 r11=0000000000000246 r12=0000000000000001 r13=0000000040000000 r14=000000001b40ef40 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 KERNELBASE!CloseHandle+0×13: 000007fe`fd651873 85c0 test eax,eax

0:002> ub 00000000`77621991 kernel32!CloseHandleImplementation+0×1e: 00000000`7762196e 83f9f4 cmp ecx,0FFFFFFF4h 00000000`77621971 0f83952e0100 jae kernel32!TlsGetValue+0×3ef0 (00000000`7763480c) 00000000`77621977 488bc3 mov rax,rbx 00000000`7762197a 2503000010 and eax,10000003h 00000000`7762197f 4883f803 cmp rax,3 00000000`77621983 0f847f8dfeff je kernel32!CloseHandleImplementation+0×56 (00000000`7760a708) 00000000`77621989 488bcb mov rcx,rbx 00000000`7762198c e81f000000 call kernel32!CloseHandle (00000000`776219b0)

Here we also check the value from the managed stack trace or from Execution Residue:

0:002> !CLRStack -a OS Thread Id: 0x1390 (2) Child SP IP Call Site 000000001b40edf8 000000007787186a [InlinedCallFrame: 000000001b40edf8] Microsoft.Win32.Win32Native.CloseHandle(IntPtr) 000000001b40edf8 000007feec720418 [InlinedCallFrame: 000000001b40edf8] Microsoft.Win32.Win32Native.CloseHandle(IntPtr) 000000001b40edd0 000007feec720418 DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr) PARAMETERS: <no data>

000000001b40eff8 000007feed8e9e03 [GCFrame: 000000001b40eff8] 000000001b40f148 000007feed8e9e03 [GCFrame: 000000001b40f148] 000000001b40f1f8 000007feed8e9e03 [HelperMethodFrame_1OBJ: 000000001b40f1f8] System.Runtime.InteropServices.SafeHandle.InternalFinalize() 000000001b40f2f0 000007feec62f7a6 System.Runtime.InteropServices.SafeHandle.Finalize() PARAMETERS: this (0x000000001b40f330) = 0×0000000002ab2d78

000000001b40f6a8 000007feed8e9d56 [DebuggerU2MCatchHandlerFrame: 000000001b40f6a8]

0:002> !dso OS Thread Id: 0x1390 (2) RSP/REG Object Name 000000001B40EEA0 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40EFD0 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F038 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F050 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F090 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F120 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F190 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F1B8 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F240 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F2F8 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F330 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F360 0000000002ab5e10 System.Threading.Thread 000000001B40F390 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F3E0 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F3F0 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle 000000001B40F430 0000000002ab58a8 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle 000000001B40F4E0 0000000002ab2d78 Microsoft.Win32.SafeHandles.SafeFileHandle

0:002> !do 0000000002ab2d78 Name: Microsoft.Win32.SafeHandles.SafeFileHandle MethodTable: 000007feec88a260 EEClass: 000007feec34d340 Size: 32(0x20) bytes File: C:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 000007feec88a338 400060d 8 System.IntPtr 1 instance dead handle 000007feec8892b8 400060e 10 System.Int32 1 instance 3 _state 000007feec887de0 400060f 14 System.Boolean 1 instance 1 _ownsHandle 000007feec887de0 4000610 15 System.Boolean 1 instance 1 _fullyInitialized

Please note that we don’t have global application flags:

0:002> !gflag Current NtGlobalFlag contents: 0x00000000

Here is the exception stack trace from a different crash dump when we enable Application Verifier:

0:002> !gflag Current NtGlobalFlag contents: 0x02000100 vrf - Enable application verifier hpa - Place heap allocations at ends of pages

0:002> k Child-SP RetAddr Call Site 00000000`24bac4a8 00000000`77cd3072 ntdll!NtWaitForSingleObject+0xa 00000000`24bac4b0 00000000`77cd32b5 ntdll!RtlReportExceptionEx+0x1d2 00000000`24bac5a0 000007fe`fa2c26fb ntdll!RtlReportException+0xb5 00000000`24bac620 00000000`77c2a5db verifier!AVrfpVectoredExceptionHandler+0x26b 00000000`24bac6b0 00000000`77c28e62 ntdll!RtlpCallVectoredHandlers+0xa8 00000000`24bac720 00000000`77c61248 ntdll!RtlDispatchException+0x22 00000000`24bace00 000007fe`fa2bae03 ntdll!KiUserExceptionDispatch+0x2e 00000000`24bad500 000007fe`fa2c268a verifier!VerifierStopMessageEx+0x6fb 00000000`24bad850 00000000`77c2a5db verifier!AVrfpVectoredExceptionHandler+0x1fa 00000000`24bad8e0 00000000`77c28e62 ntdll!RtlpCallVectoredHandlers+0xa8 00000000`24bad950 00000000`77c297c8 ntdll!RtlDispatchException+0x22 00000000`24bae030 00000000`77c612c7 ntdll!RtlRaiseException+0x22f 00000000`24bae9e0 000007fe`fa2d2386 ntdll!KiRaiseUserExceptionDispatcher+0x3a 00000000`24baeab0 000007fe`fdbd1873 verifier!AVrfpNtClose+0xbe 00000000`24baeae0 000007fe`fa2d4031 KERNELBASE!CloseHandle+0x13 00000000`24baeb10 000007fe`fa2d40cb verifier!AVrfpCloseHandleCommon+0x95 00000000`24baeb40 00000000`77a11991 verifier!AVrfpKernelbaseCloseHandle+0x23 00000000`24baeb80 000007fe`fa2d4031 kernel32!CloseHandleImplementation+0x3d 00000000`24baec90 000007fe`fa2d409c verifier!AVrfpCloseHandleCommon+0x95 *** WARNING: Unable to verify checksum for mscorlib.ni.dll 00000000`24baecc0 000007fe`e6a40418 verifier!AVrfpKernel32CloseHandle+0x2c 00000000`24baed00 000007fe`ec0e9e03 mscorlib_ni+0x580418 00000000`24baedd0 000007fe`ec0e9e7e clr!CallDescrWorkerInternal+0x83 00000000`24baee10 000007fe`ec0ec860 clr!CallDescrWorkerWithHandler+0x4a 00000000`24baee50 000007fe`ec0f1a1d clr!DispatchCallSimple+0x85 00000000`24baeee0 000007fe`ec0f19ac clr!SafeHandle::RunReleaseMethod+0x69 00000000`24baef80 000007fe`ec0f180a clr!SafeHandle::Release+0x122 00000000`24baf050 000007fe`ec24863e clr!SafeHandle::Dispose+0x36 00000000`24baf0c0 000007fe`e694f7a6 clr!SafeHandle::Finalize+0xa2 00000000`24baf220 000007fe`ec0e9d56 mscorlib_ni+0x48f7a6 00000000`24baf260 000007fe`ec283c4e clr!FastCallFinalizeWorker+0x6 00000000`24baf290 000007fe`ec283bc3 clr!MethodDesc::RequiresFullSlotNumber+0x72 00000000`24baf2d0 000007fe`ec283b0f clr!MethodTable::CallFinalizer+0xa3 00000000`24baf310 000007fe`ec1fee46 clr!SVR::CallFinalizer+0x5f 00000000`24baf350 000007fe`ec1aac5b clr!SVR::CallFinalizer+0x102 00000000`24baf410 000007fe`ec0f458c clr!WKS::GCHeap::IsPromoted+0xee 00000000`24baf450 000007fe`ec0f451a clr!Frame::Pop+0x50 00000000`24baf490 000007fe`ec0f4491 clr!COMCustomAttribute::PopSecurityContextFrame+0x192 00000000`24baf590 000007fe`ec1d1bfe clr!COMCustomAttribute::PopSecurityContextFrame+0xbd 00000000`24baf620 000007fe`ec1d1e59 clr!ManagedThreadBase_NoADTransition+0x3f 00000000`24baf680 000007fe`ec1533de clr!WKS::GCHeap::FinalizerThreadStart+0x193 00000000`24baf6c0 000007fe`fa2d4b87 clr!Thread::intermediateThreadProc+0x7d 00000000`24baf780 00000000`77a059ed verifier!AVrfpStandardThreadFunction+0x2b 00000000`24baf7c0 00000000`77c3c541 kernel32!BaseThreadInitThunk+0xd 00000000`24baf7f0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

0:002> !pe There is no current managed exception on this thread

0:002> !CLRStack OS Thread Id: 0x51e4 (2) Child SP IP Call Site 0000000024baed28 0000000077c612fa [InlinedCallFrame: 0000000024baed28] Microsoft.Win32.Win32Native.CloseHandle(IntPtr) 0000000024baed28 000007fee6a40418 [InlinedCallFrame: 0000000024baed28] Microsoft.Win32.Win32Native.CloseHandle(IntPtr) 0000000024baed00 000007fee6a40418 DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr) 0000000024baef28 000007feec0e9e03 [GCFrame: 0000000024baef28] 0000000024baf078 000007feec0e9e03 [GCFrame: 0000000024baf078] 0000000024baf128 000007feec0e9e03 [HelperMethodFrame_1OBJ: 0000000024baf128] System.Runtime.InteropServices.SafeHandle.InternalFinalize() 0000000024baf220 000007fee694f7a6 System.Runtime.InteropServices.SafeHandle.Finalize() 0000000024baf5d8 000007feec0e9d56 [DebuggerU2MCatchHandlerFrame: 0000000024baf5d8]

0:002> !dso OS Thread Id: 0x51e4 (2) RSP/REG Object Name 0000000024BAEDD0 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAEF00 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAEF68 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAEF80 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAEFC0 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF050 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF0C0 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF0E8 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF170 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF228 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF260 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF290 000000000c285e10 System.Threading.Thread 0000000024BAF2C0 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF310 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF320 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle 0000000024BAF360 000000000c2858a8 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle 0000000024BAF410 000000000c282d78 Microsoft.Win32.SafeHandles.SafeFileHandle

0:002> !CLRStack -a OS Thread Id: 0x51e4 (2) Child SP IP Call Site 0000000024baed28 0000000077c612fa [InlinedCallFrame: 0000000024baed28] Microsoft.Win32.Win32Native.CloseHandle(IntPtr) 0000000024baed28 000007fee6a40418 [InlinedCallFrame: 0000000024baed28] Microsoft.Win32.Win32Native.CloseHandle(IntPtr) 0000000024baed00 000007fee6a40418 DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr) PARAMETERS: <no data>

0000000024baef28 000007feec0e9e03 [GCFrame: 0000000024baef28] 0000000024baf078 000007feec0e9e03 [GCFrame: 0000000024baf078] 0000000024baf128 000007feec0e9e03 [HelperMethodFrame_1OBJ: 0000000024baf128] System.Runtime.InteropServices.SafeHandle.InternalFinalize() 0000000024baf220 000007fee694f7a6 System.Runtime.InteropServices.SafeHandle.Finalize() PARAMETERS: this (0x0000000024baf260) = 0×000000000c282d78

0000000024baf5d8 000007feec0e9d56 [DebuggerU2MCatchHandlerFrame: 0000000024baf5d8]

0:002> !do 0x000000000c282d78 Name: Microsoft.Win32.SafeHandles.SafeFileHandle MethodTable: 000007fee6baa260 EEClass: 000007fee666d340 Size: 32(0x20) bytes File: C:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 000007fee6baa338 400060d 8 System.IntPtr 1 instance dead handle 000007fee6ba92b8 400060e 10 System.Int32 1 instance 3 _state 000007fee6ba7de0 400060f 14 System.Boolean 1 instance 1 _ownsHandle 000007fee6ba7de0 4000610 15 System.Boolean 1 instance 1 _fullyInitialized

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns, Debugging, Software Defect Construction, x64 Windows | 1 Comment »

Crash Dump Analysis Patterns (Part 20e)

Thursday, October 30th, 2014

The set of memory dumps that prompted to introduce Insufficient Memory pattern for stack trace database also prompted to include a variant of Memory Leak pattern related to regions of virtual memory address space. We created this simple modeling application:

int _tmain(int argc, _TCHAR* argv[])
{
	int i,j;

	for (i = 1; i < 1000; ++i)
	{
		for (j = 1; j < 1000; ++j)
		{
			VirtualAlloc(NULL, 0x10000, MEM_RESERVE,
                                     PAGE_EXECUTE_READWRITE);
		}
		getc(stdin);
	}
	return 0;
}

We allocated only reserved memory regions. Committing them would probably at some stage manifest Insufficient Memory patterns for committed memory and physical memory. So we took a few consecutive memory dumps and see the ever increasing number of regions allocated at greater and greater virtual addresses:

0:000> !address [...] * 0`04070000 0`04080000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04080000 0`04090000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04090000 0`040a0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`040a0000 0`040b0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`040b0000 0`040c0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`040c0000 0`040d0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`040d0000 0`040e0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`040e0000 0`040f0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`040f0000 0`04100000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04100000 0`04110000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04110000 0`04120000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04120000 0`04130000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04130000 0`04140000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`04140000 0`04260000 0`00120000 MEM_FREE PAGE_NOACCESS Free [...]

0:000> !address [...] * 0`2eec0000 0`2eed0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2eed0000 0`2eee0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2eee0000 0`2eef0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2eef0000 0`2ef00000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef00000 0`2ef10000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef10000 0`2ef20000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef20000 0`2ef30000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef30000 0`2ef40000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef40000 0`2ef50000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef50000 0`2ef60000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef60000 0`2ef70000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef70000 0`2ef80000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef80000 0`2ef90000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2ef90000 0`2efa0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2efa0000 0`2efb0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2efb0000 0`2efc0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2efc0000 0`2efd0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2efd0000 0`2efe0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2efe0000 0`2eff0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2eff0000 0`2f000000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2f000000 0`2f010000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`2f010000 0`2f170000 0`00160000 MEM_FREE PAGE_NOACCESS Free [...]

0:000> !address [...] * 0`697f0000 0`69800000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69800000 0`69810000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69810000 0`69820000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69820000 0`69830000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69830000 0`69840000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69840000 0`69850000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69850000 0`69860000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69860000 0`69870000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69870000 0`69880000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69880000 0`69890000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`69890000 0`698a0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`698a0000 0`699e0000 0`00140000 MEM_FREE PAGE_NOACCESS Free [...]

0:000> !address [...] * 0`c08c0000 0`c08d0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c08d0000 0`c08e0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c08e0000 0`c08f0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c08f0000 0`c0900000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c0900000 0`c0910000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c0910000 0`c0920000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c0920000 0`c0930000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 0`c0930000 0`c0960000 0`00030000 MEM_FREE PAGE_NOACCESS Free [...]

0:000> !address [...] * 1`3d6a0000 1`3d6b0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d6b0000 1`3d6c0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d6c0000 1`3d6d0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d6d0000 1`3d6e0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d6e0000 1`3d6f0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d6f0000 1`3d700000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d700000 1`3d710000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d710000 1`3d720000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d720000 1`3d730000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified> * 1`3d730000 1`3d7a0000 0`00070000 MEM_FREE PAGE_NOACCESS Free [...]

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ———– Total Size ——– %ofBusy %ofTotal Free 15 7fe`c275e000 ( 7.995 Tb) 99.94% <unclassified> 80928 1`3d193000 ( 4.955 Gb) 99.86% 0.06% Image 28 0`0034b000 ( 3.293 Mb) 0.06% 0.00% Stack 6 0`00200000 ( 2.000 Mb) 0.04% 0.00% MemoryMappedFile 8 0`001af000 ( 1.684 Mb) 0.03% 0.00% TEB 2 0`00004000 ( 16.000 kb) 0.00% 0.00% PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 80936 1`3d397000 ( 4.957 Gb) 99.90% 0.06% MEM_IMAGE 29 0`0034c000 ( 3.297 Mb) 0.06% 0.00% MEM_MAPPED 8 0`001af000 ( 1.684 Mb) 0.03% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 15 7fe`c275e000 ( 7.995 Tb) 99.94% MEM_RESERVE 80926 1`3d438000 ( 4.957 Gb) 99.91% 0.06% MEM_COMMIT 47 0`0045a000 ( 4.352 Mb) 0.09% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_EXECUTE_READ 4 0`001ef000 ( 1.934 Mb) 0.04% 0.00% PAGE_READONLY 19 0`001de000 ( 1.867 Mb) 0.04% 0.00% PAGE_READWRITE 17 0`00080000 ( 512.000 kb) 0.01% 0.00% PAGE_WRITECOPY 5 0`00008000 ( 32.000 kb) 0.00% 0.00% PAGE_READWRITE|PAGE_GUARD 2 0`00005000 ( 20.000 kb) 0.00% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 1`3fac7000 7fd`bdc79000 ( 7.991 Tb) <unclassified> 0`7f0e0000 0`00f00000 ( 15.000 Mb) Image 0`77831000 0`00102000 ( 1.008 Mb) Stack 0`00170000 0`000fb000 (1004.000 kb) MemoryMappedFile 0`7efe5000 0`000fb000 (1004.000 kb) TEB 7ff`fffdc000 0`00002000 ( 8.000 kb) PEB 7ff`fffd3000 0`00001000 ( 4.000 kb)

Examination of such regions for Execution Residue such as Module Hint may point into further troubleshooting directions especially if live debugging is not possible.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Models, Software Defect Construction, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 13k)

Monday, October 27th, 2014

While working on Insufficient Memory pattern for stack trace database we noticed the expansion of certain memory regions. Of course, after some time expanding region consumes remaining free or reserved space available before some other region. Generalizing from this we may say there can be Insufficient Memory pattern variant for any region expanding region. Region expansion may also be implemented via its move into some over position in memory virtual address space. This movement also has its limits. for example, we created this modeling application and found out it stops reallocating memory long before it reaches 2,000,000,000 byte size:

int _tmain(int argc, _TCHAR* argv[])
{
	int i = 100000000;

	void *p = malloc(i);

	for (i = 200000000; i < 2000000000; i+=100000000)
	{
		p = realloc(p, i);
		getc(stdin);
	}

	return 0;
}

We took memory dumps after each loop iteration and after 6 or 8 iterations the memory size was constant and there were no further reallocations:

0:000> !heap -s [...] Virtual block: 0000000006370000 - 0000000006370000 (size 0000000000000000) [...]

0:000> !address [...] + 0`00550000 0`06370000 0`05e20000 MEM_FREE PAGE_NOACCESS Free + 0`06370000 0`1222d000 0`0bebd000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block] + 0`1222d000 0`77710000 0`654e3000 MEM_FREE PAGE_NOACCESS Free + 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”] […]

0:000> !heap -s [...] Virtual block: 0000000012230000 - 0000000012230000 (size 0000000000000000) [...]

0:000> !address [...] + 0`005d0000 0`12230000 0`11c60000 MEM_FREE PAGE_NOACCESS Free + 0`12230000 0`2404b000 0`11e1b000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block] + 0`2404b000 0`77710000 0`536c5000 MEM_FREE PAGE_NOACCESS Free + 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”] […]

0:000> !heap -s [...] Virtual block: 0000000024050000 - 0000000024050000 (size 0000000000000000) [...]

0:000> !address [...] + 0`00590000 0`24050000 0`23ac0000 MEM_FREE PAGE_NOACCESS Free + 0`24050000 0`3bdc9000 0`17d79000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block] + 0`3bdc9000 0`77710000 0`3b947000 MEM_FREE PAGE_NOACCESS Free + 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”] […]

We skip a few iterations and finally come to a region that will not move and not increase:

0:000> !heap -s [...] Virtual block: 0000000041d30000 - 0000000041d30000 (size 0000000000000000) [...]

0:000> !address [...] + 0`006c0000 0`41d30000 0`41670000 MEM_FREE PAGE_NOACCESS Free + 0`41d30000 0`6b8c3000 0`29b93000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block] + 0`6b8c3000 0`77710000 0`0be4d000 MEM_FREE PAGE_NOACCESS Free + 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”] […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Models, Software Defect Construction, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 13j)

Monday, October 27th, 2014

Once we have seen a sequence of process memory dumps with the largest one almost 4GB. They were all saved from the process with growing memory consumption from 200MB initially. Initially we suspected process heap Memory Leak. However, heap statistics (!heap -s) was normal. There were not even large block allocations. The dumps were also supplied with UMDH logs but their difference only showed Memory Fluctuation and not increase. Stack Trace Collection revealed one Spiking Thread was logging a heap allocation into user mode stack trace database. We could also see that it was Distributed Spike. Inspection of address space showed large number of sequential regions of the same size with Stack Trace Database entries inside. So we concluded that it was stack trace logging Instrumentation Side Effect and advised to limit stack backtrace size in gflags.exe.

To make sure we understood that problem correctly we decided to model it. We didn’t come to the same results probably due to different logging implementation but memory dumps clearly show the possibility of Insufficient Memory pattern variant. Here’s the source code:

void foo20 (int size)
{
    free(malloc(size));
}

#define FOO(x,y) void foo##x (int size) { foo##y(size); }

FOO(19,20)
FOO(18,19)
FOO(17,18)
FOO(16,17)
FOO(15,16)
FOO(14,15)
FOO(13,14)
FOO(12,13)
FOO(11,12)
FOO(10,11)
FOO(9,10)
FOO(8,9)
FOO(7,8)
FOO(6,7)
FOO(5,6)
FOO(4,5)
FOO(3,4)
FOO(2,3)
FOO(1,2)

typedef void (*PFN) (int);

#define ARRSZ 20

PFN pfnArr[ARRSZ]  = {foo1, foo2, foo3, foo4, foo5, foo6, foo7,
    foo8, foo9, foo10, foo11, foo12, foo13, foo14,
    foo15, foo16, foo17, foo18, foo19, foo20};

int _tmain(int argc, _TCHAR* argv[])
{
    int i;

    for (i = 1; i < 1000000000; ++i) 
    {
        pfnArr[i%ARRSZ](i);
    }

    Sleep(-1); 
    return 0;
}

It allocates and then freezes heap entries of different size from 1 byte to 1,000,000,000 bytes all with different 20 possible stack traces. We choose different stack traces to increase the number of different {size, stack backtrace} pairs as several allocation of similar size having the same stack trace may be recorded only once in the database. We emulate different stack traces by calling different entries in pfnArr. Each call then leads to foo20 but the resulting stack trace depth is different. We also enabled “Create user mode stack trace database” checkbox in gflags.exe for our application called AllocFree.exe.

Then we see the expansion of Stack Trace Database regions (addresses are different because memory dumps were taken from different application runs):

0:000> !address [...] + 0`00240000 0`00312000 0`000d2000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database] 0`00312000 0`01a37000 0`01725000 MEM_PRIVATE MEM_RESERVE Other [Stack Trace Database] 0`01a37000 0`01a40000 0`00009000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database]

0:000> !address [...] + 0`001b0000 0`0188c000 0`016dc000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database] 0`0188c000 0`0188d000 0`00001000 MEM_PRIVATE MEM_RESERVE Other [Stack Trace Database] 0`0188d000 0`019b0000 0`00123000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database]

Heap stays the same:

0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key                   : 0x000000f841c4f9c0
Termination on corruption : ENABLED
           Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock
  Fast
                             (k)     (k)    (k)     (k) length      blocks cont.
 heap
-------------------------------------------------------------------------------
------
0000000001a40000 08000002    4096   1444   4096   1164     4     3    0      0
   LFH
External fragmentation  80 % (4 free blocks)
0000000000010000 08008000      64      4     64      1     1     1    0      0
0000000000020000 08008000      64     64     64     61     1     1    0      0
-------------------------------------------------------------------------------
------

0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key                   : 0x000000473a639107
Termination on corruption : ENABLED
           Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock
  Fast
                             (k)     (k)    (k)     (k) length      blocks cont.
 heap
-------------------------------------------------------------------------------
------
00000000019c0000 08000002    4096   1444   4096   1164     4     3    0      0
   LFH
External fragmentation  80 % (4 free blocks)
0000000000010000 08008000      64      4     64      1     1     1    0      0
0000000000020000 08008000      64     64     64     61     1     1    0      0
-------------------------------------------------------------------------------
------

But we see the thread consuming much CPU and caught while logging stack backtrace:

0:000> kc Call Site ntdll!RtlpStdLogCapturedStackTrace ntdll!RtlStdLogStackTrace ntdll!RtlLogStackBackTraceEx ntdll!RtlpAllocateHeap ntdll!RtlAllocateHeap AllocFree!_heap_alloc AllocFree!malloc AllocFree!foo20 AllocFree!foo19 AllocFree!foo18 AllocFree!foo17 AllocFree!foo16 AllocFree!foo15 AllocFree!foo14 AllocFree!foo13 AllocFree!foo12 AllocFree!foo11 AllocFree!foo10 AllocFree!foo9 AllocFree!foo8 AllocFree!foo7 AllocFree!foo6 AllocFree!foo5 AllocFree!foo4 AllocFree!foo3 AllocFree!foo2 AllocFree!foo1 AllocFree!wmain AllocFree!__tmainCRTStartup kernel32!BaseThreadInitThunk ntdll!RtlUserThreadStart

0:000> !runaway f
 User Mode Time
  Thread       Time
   0:53b8      0 days 3:22:02.354
 Kernel Mode Time
  Thread       Time
   0:53b8      0 days 0:20:39.022
 Elapsed Time
  Thread       Time
   0:53b8      0 days 10:11:23.596

If we dump some portion of the region we see recorded stack backtraces:

0:000> dps 0`0188c000-200 L200/8 00000000`0188be00 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd 00000000`0188be08 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c 00000000`0188be10 00000001`3fcc13cb AllocFree!malloc+0x5b 00000000`0188be18 00000001`3fcc1015 AllocFree!foo20+0x15 00000000`0188be20 00000001`3fcc1041 AllocFree!foo19+0x11 00000000`0188be28 00000001`3fcc1061 AllocFree!foo18+0x11 00000000`0188be30 00000001`3fcc12e3 AllocFree!wmain+0x53 00000000`0188be38 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144 00000000`0188be40 00000000`777259ed kernel32!BaseThreadInitThunk+0xd 00000000`0188be48 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d 00000000`0188be50 00000000`0188b1d0 00000000`0188be58 0009457d`00024fff 00000000`0188be60 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd 00000000`0188be68 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c 00000000`0188be70 00000001`3fcc13cb AllocFree!malloc+0x5b 00000000`0188be78 00000001`3fcc1015 AllocFree!foo20+0x15 00000000`0188be80 00000001`3fcc1041 AllocFree!foo19+0x11 00000000`0188be88 00000001`3fcc12e3 AllocFree!wmain+0x53 00000000`0188be90 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144 00000000`0188be98 00000000`777259ed kernel32!BaseThreadInitThunk+0xd 00000000`0188bea0 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d 00000000`0188bea8 00000000`00000000 00000000`0188beb0 00000000`0188b230 00000000`0188beb8 0008457e`00023fff 00000000`0188bec0 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd 00000000`0188bec8 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c 00000000`0188bed0 00000001`3fcc13cb AllocFree!malloc+0x5b 00000000`0188bed8 00000001`3fcc1015 AllocFree!foo20+0x15 00000000`0188bee0 00000001`3fcc12e3 AllocFree!wmain+0x53 00000000`0188bee8 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144 00000000`0188bef0 00000000`777259ed kernel32!BaseThreadInitThunk+0xd 00000000`0188bef8 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d 00000000`0188bf00 00000000`0188b280 00000000`0188bf08 001b457f`0002dfff 00000000`0188bf10 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd 00000000`0188bf18 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c 00000000`0188bf20 00000001`3fcc13cb AllocFree!malloc+0x5b 00000000`0188bf28 00000001`3fcc1015 AllocFree!foo20+0x15 00000000`0188bf30 00000001`3fcc1041 AllocFree!foo19+0x11 00000000`0188bf38 00000001`3fcc1061 AllocFree!foo18+0x11 00000000`0188bf40 00000001`3fcc1081 AllocFree!foo17+0x11 00000000`0188bf48 00000001`3fcc10a1 AllocFree!foo16+0x11 00000000`0188bf50 00000001`3fcc10c1 AllocFree!foo15+0x11 00000000`0188bf58 00000001`3fcc10e1 AllocFree!foo14+0x11 00000000`0188bf60 00000001`3fcc1101 AllocFree!foo13+0x11 00000000`0188bf68 00000001`3fcc1121 AllocFree!foo12+0x11 00000000`0188bf70 00000001`3fcc1141 AllocFree!foo11+0x11 00000000`0188bf78 00000001`3fcc1161 AllocFree!foo10+0x11 00000000`0188bf80 00000001`3fcc1181 AllocFree!foo9+0x11 00000000`0188bf88 00000001`3fcc11a1 AllocFree!foo8+0x11 00000000`0188bf90 00000001`3fcc11c1 AllocFree!foo7+0x11 00000000`0188bf98 00000001`3fcc11e1 AllocFree!foo6+0x11 00000000`0188bfa0 00000001`3fcc1201 AllocFree!foo5+0x11 00000000`0188bfa8 00000001`3fcc1221 AllocFree!foo4+0x11 00000000`0188bfb0 00000001`3fcc1241 AllocFree!foo3+0x11 00000000`0188bfb8 00000001`3fcc1261 AllocFree!foo2+0x11 00000000`0188bfc0 00000001`3fcc1281 AllocFree!foo1+0x11 00000000`0188bfc8 00000001`3fcc12e3 AllocFree!wmain+0x53 00000000`0188bfd0 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144 00000000`0188bfd8 00000000`777259ed kernel32!BaseThreadInitThunk+0xd 00000000`0188bfe0 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d 00000000`0188bfe8 00000000`00000000 00000000`0188bff0 00000000`00000000 00000000`0188bff8 00000000`00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Models, Software Defect Construction, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 25d)

Tuesday, October 21st, 2014

Some troubleshooting and debugging techniques involve saving every Stack Trace that leads to a specific action such as a memory allocation of opening of a resource handle to be saved in some region in memory, called stack trace database. Typical pattern usage examples include Process Heap Memory Leak, Insufficient Memory due to Handle Leak. Typical entry in such a database consists of return addresses saved during function calls (which may be Truncated Stack Trace):

00000000`00325da0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325da8 00000001`3fd72239 AllocFree!_ioinit+0×2cd 00000000`00325db0 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5 00000000`00325db8 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325dc0 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d

0:001> ub 00000001`3fd72239 AllocFree!_ioinit+0×2af: 00000001`3fd7221b cmp eax,3 00000001`3fd7221e jne AllocFree!_ioinit+0×2be (00000001`3fd7222a) 00000001`3fd72220 movsx eax,byte ptr [rbx+8] 00000001`3fd72224 or eax,8 00000001`3fd72227 mov byte ptr [rbx+8],al 00000001`3fd7222a lea rcx,[rbx+10h] 00000001`3fd7222e mov edx,0FA0h 00000001`3fd72233 call qword ptr [AllocFree!_imp_InitializeCriticalSectionAndSpinCount (00000001`3fd78090)

This slightly differs from ‘k’-style stack trace format where the return address belongs to the function on the next line if moving downwards:

0:000> k Child-SP RetAddr Call Site 00000000`002ff9f8 000007fe`fd5e1203 ntdll!ZwDelayExecution+0xa 00000000`002ffa00 00000001`3fd71018 KERNELBASE!SleepEx+0xab 00000000`002ffaa0 00000001`3fd71194 AllocFree!wmain+0×18 00000000`002ffad0 00000000`773759ed AllocFree!__tmainCRTStartup+0×144 00000000`002ffb10 00000000`774ac541 kernel32!BaseThreadInitThunk+0xd 00000000`002ffb40 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub 00000001`3fd71194 AllocFree!__tmainCRTStartup+0×11b: 00000001`3fd7116b je AllocFree!__tmainCRTStartup+0×124 (00000001`3fd71174) 00000001`3fd7116d mov ecx,eax 00000001`3fd7116f call AllocFree!_amsg_exit (00000001`3fd718ec) 00000001`3fd71174 mov r8,qword ptr [AllocFree!_wenviron (00000001`3fd80868)] 00000001`3fd7117b mov qword ptr [AllocFree!__winitenv (00000001`3fd80890)],r8 00000001`3fd71182 mov rdx,qword ptr [AllocFree!__wargv (00000001`3fd80858)] 00000001`3fd71189 mov ecx,dword ptr [AllocFree!__argc (00000001`3fd8084c)] 00000001`3fd7118f call AllocFree!wmain (00000001`3fd71000)

Sometimes we can see such traces as Execution Residue inside a stack or some other region. If user mode stack trace database is enabled in gflags.exe we might be able to dump the specific database region:

0:001> !gflag Current NtGlobalFlag contents: 0x00001000 ust - Create user mode stack trace database

0:001> !address [...] BaseAddress EndAddress+1 RegionSize Type State Protect Usage ------------------------------------------------------------------------------------------------------------------------ [...] + 0`00300000 0`00326000 0`00026000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database] 0`00326000 0`01aff000 0`017d9000 MEM_PRIVATE MEM_RESERVE Other [Stack Trace Database] 0`01aff000 0`01b00000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database] […]

0:001> dps 0`00326000-1000 0`00326000 […] 00000000`003257e0 00000000`00000000 00000000`003257e8 00030001`00001801 00000000`003257f0 00000000`774c34eb ntdll!LdrpInitializeProcess+0×7e6 00000000`003257f8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325800 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325808 00000000`00000000 00000000`00325810 00000000`00000000 00000000`00325818 00030002`00001801 00000000`00325820 00000000`774c3511 ntdll!LdrpInitializeProcess+0×80c 00000000`00325828 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325830 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325838 00000000`00000000 00000000`00325840 00000000`00000000 00000000`00325848 00040003`00001801 00000000`00325850 00000000`774bda86 ntdll!RtlCreateHeap+0×506 00000000`00325858 00000000`774c3557 ntdll!LdrpInitializeProcess+0×851 00000000`00325860 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325868 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325870 00000000`00000000 00000000`00325878 00050004`00002801 00000000`00325880 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a 00000000`00325888 00000000`774bdaee ntdll!RtlCreateHeap+0×56e 00000000`00325890 00000000`774c3557 ntdll!LdrpInitializeProcess+0×851 00000000`00325898 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`003258a0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`003258a8 00000000`00000000 00000000`003258b0 00000000`00000000 00000000`003258b8 00030005`00001801 00000000`003258c0 00000000`774c359e ntdll!LdrpInitializeProcess+0×902 00000000`003258c8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`003258d0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`003258d8 00000000`00000000 00000000`003258e0 00000000`00000000 00000000`003258e8 00030006`00001801 00000000`003258f0 00000000`774c35af ntdll!LdrpInitializeProcess+0×913 00000000`003258f8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325900 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325908 00000000`00000000 00000000`00325910 00000000`00000000 00000000`00325918 00090007`00004801 00000000`00325920 00000000`774bda86 ntdll!RtlCreateHeap+0×506 00000000`00325928 00000000`774c47ff ntdll!CsrpConnectToServer+0×41f 00000000`00325930 00000000`774c43c5 ntdll!CsrClientConnectToServer+0×230 00000000`00325938 000007fe`fd5ee232 KERNELBASE!KernelBaseDllInitialize+0×148 00000000`00325940 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325948 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325950 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325958 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325960 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325968 00000000`00000000 00000000`00325970 00000000`00000000 00000000`00325978 000a0008`00004801 00000000`00325980 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a 00000000`00325988 00000000`774bdaee ntdll!RtlCreateHeap+0×56e 00000000`00325990 00000000`774c47ff ntdll!CsrpConnectToServer+0×41f 00000000`00325998 00000000`774c43c5 ntdll!CsrClientConnectToServer+0×230 00000000`003259a0 000007fe`fd5ee232 KERNELBASE!KernelBaseDllInitialize+0×148 00000000`003259a8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`003259b0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`003259b8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`003259c0 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`003259c8 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`003259d0 00000000`00000000 00000000`003259d8 00080009`00003801 00000000`003259e0 000007fe`fd5edf81 KERNELBASE!NlsProcessInitialize+0×11 00000000`003259e8 000007fe`fd604439 KERNELBASE!BaseNlsDllInitialize+0×29 00000000`003259f0 000007fe`fd5ee446 KERNELBASE!KernelBaseDllInitialize+0×40c 00000000`003259f8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325a00 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325a08 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325a10 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325a18 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325a20 00000000`00000000 00000000`00325a28 0008000a`00003801 00000000`00325a30 000007fe`fd5edfa0 KERNELBASE!NlsProcessInitialize+0×30 00000000`00325a38 000007fe`fd604439 KERNELBASE!BaseNlsDllInitialize+0×29 00000000`00325a40 000007fe`fd5ee446 KERNELBASE!KernelBaseDllInitialize+0×40c 00000000`00325a48 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325a50 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325a58 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325a60 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325a68 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325a70 00000000`00000000 00000000`00325a78 0007000b`00003801 00000000`00325a80 000007fe`fd604a21 KERNELBASE!BasepInitComputerNameCache+0×11 00000000`00325a88 000007fe`fd603d20 KERNELBASE!KernelBaseDllInitialize+0×419 00000000`00325a90 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325a98 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325aa0 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325aa8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325ab0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325ab8 00000000`00000000 00000000`00325ac0 00000000`00000000 00000000`00325ac8 0006000c`00002801 00000000`00325ad0 00000000`77375699 kernel32!BaseDllInitialize+0×2f9 00000000`00325ad8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325ae0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325ae8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325af0 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325af8 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325b00 00000000`00000000 00000000`00325b08 0007000d`00003801 00000000`00325b10 00000000`773771f7 kernel32!InitializeConsoleConnectionInfo+0xe7 00000000`00325b18 00000000`773756ae kernel32!BaseDllInitialize+0×30e 00000000`00325b20 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325b28 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325b30 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325b38 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325b40 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325b48 00000000`00000000 00000000`00325b50 00000000`00000000 00000000`00325b58 0009000e`00004801 00000000`00325b60 00000000`774bda86 ntdll!RtlCreateHeap+0×506 00000000`00325b68 00000000`773787f7 kernel32!ConsoleConnect+0×1d7 00000000`00325b70 00000000`773770de kernel32!ConnectConsoleInternal+0×147 00000000`00325b78 00000000`773756fe kernel32!BaseDllInitialize+0×35e 00000000`00325b80 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325b88 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325b90 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325b98 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325ba0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325ba8 00000000`00000000 00000000`00325bb0 00000000`00000000 00000000`00325bb8 000a000f`00004801 00000000`00325bc0 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a 00000000`00325bc8 00000000`774bdaee ntdll!RtlCreateHeap+0×56e 00000000`00325bd0 00000000`773787f7 kernel32!ConsoleConnect+0×1d7 00000000`00325bd8 00000000`773770de kernel32!ConnectConsoleInternal+0×147 00000000`00325be0 00000000`773756fe kernel32!BaseDllInitialize+0×35e 00000000`00325be8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325bf0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325bf8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325c00 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325c08 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325c10 00000000`00000000 00000000`00325c18 00060010`00002801 00000000`00325c20 00000000`773757dc kernel32!BaseDllInitialize+0×43c 00000000`00325c28 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325c30 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325c38 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325c40 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325c48 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325c50 00000000`00000000 00000000`00325c58 00060011`00002801 00000000`00325c60 00000000`7737582c kernel32!BaseDllInitialize+0×48c 00000000`00325c68 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325c70 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325c78 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325c80 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325c88 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325c90 00000000`00000000 00000000`00325c98 00060012`0000280e 00000000`00325ca0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325ca8 00000001`3fd7319f AllocFree!_mtinitlocks+0×43 00000000`00325cb0 00000001`3fd717fc AllocFree!_mtinit+0×10 00000000`00325cb8 00000001`3fd710e4 AllocFree!__tmainCRTStartup+0×94 00000000`00325cc0 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325cc8 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325cd0 00000000`00000000 00000000`00325cd8 000b0013`00005801 00000000`00325ce0 00000000`774c1131 ntdll!RtlpActivateLowFragmentationHeap+0×181 00000000`00325ce8 00000000`774c0f97 ntdll!RtlpPerformHeapMaintenance+0×27 00000000`00325cf0 00000000`774c0f5b ntdll!RtlpAllocateHeap+0×1819 00000000`00325cf8 00000000`774d34d8 ntdll!RtlAllocateHeap+0×16c 00000000`00325d00 00000000`774a9300 ntdll!RtlInitializeCriticalSectionAndSpinCount+0×183 00000000`00325d08 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325d10 00000001`3fd7319f AllocFree!_mtinitlocks+0×43 00000000`00325d18 00000001`3fd717fc AllocFree!_mtinit+0×10 00000000`00325d20 00000001`3fd710e4 AllocFree!__tmainCRTStartup+0×94 00000000`00325d28 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325d30 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325d38 00000000`00000000 00000000`00325d40 00000000`00000000 00000000`00325d48 00070014`00003801 00000000`00325d50 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325d58 00000001`3fd7312f AllocFree!_mtinitlocknum+0×8f 00000000`00325d60 00000001`3fd72ff7 AllocFree!_lock+0×23 00000000`00325d68 00000001`3fd71f9b AllocFree!_ioinit+0×2f 00000000`00325d70 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5 00000000`00325d78 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325d80 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325d88 00000000`00000000 00000000`00325d90 00000000`00000000 00000000`00325d98 00050015`00002803 00000000`00325da0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325da8 00000001`3fd72239 AllocFree!_ioinit+0×2cd 00000000`00325db0 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5 00000000`00325db8 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325dc0 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325dc8 00000000`00000000 00000000`00325dd0 00000000`00000000 […]

This database corresponds to this simple program:

int _tmain(int argc, _TCHAR* argv[])
{
    free(malloc(256));
    Sleep(-1);
    return 0;
}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 25c)

Sunday, October 12th, 2014

Sometimes threads related to file system operations may be blocked with not easily recognizable 3rd-party Top Module with only OS vendor modules such as NTFS or fltmgr present:

nt!KiSwapContext+0x7a nt!KiCommitThreadWait+0x1d2 nt!KeWaitForSingleObject+0x19f nt!FsRtlCancellableWaitForMultipleObjects+0x5e nt!FsRtlCancellableWaitForSingleObject+0x27 fltmgr! ?? ::FNODOBFM::`string’+0×2bfa fltmgr!FltpCreate+0×2a9 nt!IopParseDevice+0×14d3 nt!ObpLookupObjectName+0×588 nt!ObOpenObjectByName+0×306 nt!IopCreateFile+0×2bc nt!NtCreateFile+0×78 nt!KiSystemServiceCopyEnd+0×13 ntdll!NtCreateFile+0xa […]

We see the same modules in I/O Request Stack Trace from the thread IRP. But because we see filter manager involved there may be some 3rd-party file system filters involved. Such filters are called before a device processes a request and also upon the completion of the request. There may be different filter callbacks registered for each case and they form a similar structure like I/O stack locations (we call this pattern Filter Stack Trace):

If one of such filters is blocked in a wait chain this may not be visible on I/O request or thread stacks because of possible asynchronous processing. But we may use !fltkd.irpctrl debugging extension command to examine the IRP context:

0: kd> !irp fffffa80162aa230 cmd flg cl Device File Completion-Context [...] [ 0, 0] 0 0 fffffa800cb28030 00000000 fffff880012048f0-fffffa8016f64010 \FileSystem\Ntfs fltmgr!FltpSynchronizedOperationCompletion Args: 00000000 00000000 00000000 00000000 >[ 0, 0] 0 1 fffffa800ca00890 fffffa801060d070 00000000-00000000 pending \FileSystem\FltMgr Args: fffff88014450868 02000060 00000006 00000000

0: kd> !fltkd.irpctrl fffffa8016f64010 [...] Cmd IrpFl OpFl CmpFl Instance FileObjt Completion-Context Node Adr --------- -------- ----- ----- -------- -------- ------------------ -------- [0,0] 00000884 00 0000 fffffa800d29c010 fffffa801060d070 fffff8800518b474-0000000000000000 fffffa8016f641e0 ("luafv","luafv") luafv!LuafvPostCreate Args: fffff88014450868 0000000002000060 0000000000000006 0000000000000000 0000000000000000 0000000000000000 >[0,0] 00000884 00 0000 fffffa800e8051d0 fffffa801060d070 fffff88006808440-0000000000000000 fffffa8016f64160 (”3rdPartyFilter”,”3rdPartyFilter Instance”) FilterA!FltDriver_PostOperationCallback Args: fffff88014450868 0000000002000060 0000000000000006 0000000000000000 0000000000000000 0000000000000000 […]

So we see that FilterA module may be involved in blocking the thread (Blocking Module pattern extended to I/O request and filter stack traces).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Archive for the ‘x64 Windows’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta