Software Diagnostics Library

Crash Dump Analysis Patterns (Part 14, Linux)

December 14th, 2015

This is a variant of Spiking Thread pattern previously described for Mac OS X and Windows platforms:

(gdb) info threads Id Target Id Frame 6 LWP 3712 0x00000000004329d1 in nanosleep () 5 LWP 3717 0×00000000004007a3 in isnan () 4 LWP 3716 0×00000000004329d1 in nanosleep () 3 LWP 3715 0×00000000004329d1 in nanosleep () 2 LWP 3714 0×00000000004329d1 in nanosleep () * 1 LWP 3713 0×00000000004329d1 in nanosleep ()

We notice a non-waiting thread and switch to it:

(gdb) thread 5 [Switching to thread 5 (LWP 3717)] #0 0x00000000004007a3 in isnan ()

(gdb) bt #0 0×00000000004007a3 in isnan () #1 0×0000000000400743 in sqrt () #2 0×0000000000400528 in procB () #3 0×0000000000400639 in bar_five () #4 0×0000000000400649 in foo_five () #5 0×0000000000400661 in thread_five () #6 0×0000000000403e30 in start_thread () #7 0×0000000000435089 in clone () #8 0×0000000000000000 in ?? ()

If we disassemble the return address for procB function to come back from sqrt call we see an infinite loop:

(gdb) disassemble 0x400528 Dump of assembler code for function procB: 0x0000000000400500 <+0>: push %rbp 0x0000000000400501 <+1>: mov %rsp,%rbp 0x0000000000400504 <+4>: sub $0x20,%rsp 0x0000000000400508 <+8>: movabs $0x3fd5555555555555,%rax 0x0000000000400512 <+18>: mov %rax,-0x8(%rbp) 0×0000000000400516 <+22>: mov -0×8(%rbp),%rax 0×000000000040051a <+26>: mov %rax,-0×18(%rbp) 0×000000000040051e <+30>: movsd -0×18(%rbp),%xmm0 0×0000000000400523 <+35>: callq 0×400710 <sqrt> 0×0000000000400528 <+40>: movsd %xmm0,-0×18(%rbp) 0×000000000040052d <+45>: mov -0×18(%rbp),%rax 0×0000000000400531 <+49>: mov %rax,-0×8(%rbp) 0×0000000000400535 <+53>: jmp 0×400516 <procB+22> End of assembler dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Assembly Language, Core Dump Analysis, Crash Dump Patterns, Linux Crash Corner, x64 Linux | No Comments »

Crash Dump Analysis Patterns (Part 77, Linux)

December 14th, 2015

This is a Linux variant of C++ Exception pattern previously described for Mac OS X and Windows platforms:

(gdb) bt #0 0x00007f0a1d0e5165 in *__GI_raise () at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007f0a1d0e83e0 in *__GI_abort () at abort.c:92 #2 0x00007f0a1db5789d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007f0a1db55996 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007f0a1db559c3 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007f0a1db55bee in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x0000000000400dcf in procB() () #7 0x0000000000400e26 in procA() () #8 0x0000000000400e88 in procNH() () #9 0x0000000000400ea8 in bar_one() () #10 0x0000000000400eb3 in foo_one() () #11 0x0000000000400ec6 in thread_one(void*) () #12 0x00007f0a1d444b50 in start_thread () #13 0x00007f0a1d18e95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #14 0x0000000000000000 in ?? ()

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Patterns, Linux Crash Corner, x64 Linux | No Comments »

Crash Dump Analysis Patterns (Part 180, Linux)

December 14th, 2015

This is Linux variant of Paratext pattern for Mac OS X. Because of debugger tool limitations additional software logs and the output of other tools may help in memory dump analysis. Typical examples of such pattern usage can be the list of modules with version and path info, application crash specific information from instrumentation tools such as valgrind, memory region names with attribution and boundaries, and CPU usage information. For example, top and pmap commands output:

14039: ./App1.shared 0000000000400000 4K r-x-- /home/training/ALCDA/App1/App1.shared 0000000000600000 4K rw--- /home/training/ALCDA/App1/App1.shared 0000000000611000 132K rw--- [ anon ] 00007fe8999a6000 4K ----- [ anon ] 00007fe8999a7000 8192K rw--- [ anon ] 00007fe89a1a7000 4K ----- [ anon ] 00007fe89a1a8000 8192K rw--- [ anon ] 00007fe89a9a8000 4K ----- [ anon ] 00007fe89a9a9000 8192K rw--- [ anon ] 00007fe89b1a9000 4K ----- [ anon ] 00007fe89b1aa000 8192K rw--- [ anon ] 00007fe89b9aa000 4K ----- [ anon ] 00007fe89b9ab000 8192K rw--- [ anon ] 00007fe89c1ab000 1540K r-x-- /lib/x86_64-linux-gnu/libc-2.13.so 00007fe89c32c000 2048K ----- /lib/x86_64-linux-gnu/libc-2.13.so 00007fe89c52c000 16K r---- /lib/x86_64-linux-gnu/libc-2.13.so 00007fe89c530000 4K rw--- /lib/x86_64-linux-gnu/libc-2.13.so 00007fe89c531000 20K rw--- [ anon ] 00007fe89c536000 92K r-x-- /lib/x86_64-linux-gnu/libpthread-2.13.so 00007fe89c54d000 2044K ----- /lib/x86_64-linux-gnu/libpthread-2.13.so 00007fe89c74c000 4K r---- /lib/x86_64-linux-gnu/libpthread-2.13.so 00007fe89c74d000 4K rw--- /lib/x86_64-linux-gnu/libpthread-2.13.so 00007fe89c74e000 16K rw--- [ anon ] 00007fe89c752000 128K r-x-- /lib/x86_64-linux-gnu/ld-2.13.so 00007fe89c966000 12K rw--- [ anon ] 00007fe89c96f000 8K rw--- [ anon ] 00007fe89c971000 4K r---- /lib/x86_64-linux-gnu/ld-2.13.so 00007fe89c972000 4K rw--- /lib/x86_64-linux-gnu/ld-2.13.so 00007fe89c973000 4K rw--- [ anon ] 00007ffd458c1000 132K rw--- [ stack ] 00007ffd459e9000 4K r-x-- [ anon ] ffffffffff600000 4K r-x-- [ anon ] total 47208K

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology, GDB Annoyances, GDB for WinDbg Users, Linux Crash Corner, Software Diagnostics, Software Trace Analysis, x64 Linux | No Comments »

Crash Dump Analysis Patterns (Part 234)

November 30th, 2015

Sometimes we identify a function parameter that caused abnormal behavior, for example, Invalid Pointer access violation and want to see from which module or function it was probably originated:

0:009> kvL # ChildEBP RetAddr Args to Child 00 0381f048 74e715e9 00000002 0381f098 00000001 ntdll!NtWaitForMultipleObjects+0x15 01 0381f0e4 76ce19fc 0381f098 0381f10c 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 02 0381f12c 76ce41d8 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 03 0381f148 76d08074 00000002 0381f17c 00000000 kernel32!WaitForMultipleObjects+0x18 04 0381f1b4 76d07f33 0381f294 00000001 00000001 kernel32!WerpReportFaultInternal+0x186 05 0381f1c8 76d07828 0381f294 00000001 0381f264 kernel32!WerpReportFault+0x70 06 0381f1d8 76d077a7 0381f294 00000001 9a131a15 kernel32!BasepReportFault+0x20 07 0381f264 772574ff 00000000 772573dc 00000000 kernel32!UnhandledExceptionFilter+0x1af 08 0381f26c 772573dc 00000000 0381ffd4 7720c550 ntdll!__RtlUserThreadStart+0x62 09 0381f280 77257281 00000000 00000000 00000000 ntdll!_EH4_CallFilterFunc+0x12 0a 0381f2a8 7723b499 fffffffe 0381ffc4 0381f3e4 ntdll!_except_handler4+0x8e 0b 0381f2cc 7723b46b 0381f394 0381ffc4 0381f3e4 ntdll!ExecuteHandler2+0x26 0c 0381f2f0 7723b40e 0381f394 0381ffc4 0381f3e4 ntdll!ExecuteHandler+0x24 0d 0381f37c 771f0133 0181f394 0381f3e4 0381f394 ntdll!RtlDispatchException+0x127 0e 0381f37c 00010101 0181f394 0381f3e4 0381f394 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0381f3e4) WARNING: Frame IP not in any known module. Following frames may be wrong. 0f 0381f844 00df3d1e 064bf018 0b925970 06501dd8 0×10101 10 0381f860 00ca0a15 064bf018 0b925970 0c86a949 ModuleA!Bar+0×3e 11 0381f888 00ca07d4 064bf018 0b925970 09584580 ModuleB!Foo2+0×75 12 0381f8e4 00ca0ae3 0b925970 00000000 09584008 ModuleB!Foo1+0xb4 […]

Here we backtrack the invalid function call pointer to a function argument parameter:

0:009> .cxr 0381f3e4 eax=06501dd8 ebx=0381f8c4 ecx=0b925970 edx=0b925527 esi=064bf018 edi=0381f898 eip=00010101 esp=0381f848 ebp=0c86a949 iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 00010101 1f pop ds

0:009> k 1 *** Stack trace for last set context - .thread/.cxr resets it # ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 00 0381f844 00df3d1e 0×10101

0:009> ub 00df3d1e ModuleA!Bar+0x2e: 00df3d0a mov ecx,dword ptr [esp+10h] 00df3d0e mov edi,dword ptr [esp+18h] 00df3d12 push edi 00df3d13 push eax 00df3d14 mov edx,dword ptr [ecx] 00df3d16 push ecx 00df3d17 push esi 00df3d18 mov edx,dword ptr [edx+4] 00df3d1b call dword ptr [edx+10h]

The value of EDX comes from EDX+4 which comes from ECX pointer value which was probably one of function parameters:

0:009> dp ecx L1 0b925970 0b925978

0:009> dp poi(ecx)+4 L1 0b92597c 0b925527

0:009> dc 0b925527 0b925527 20202020 80202020 01010101 01010101 ……… 0b925537 00010101 01010101 01010000 01000101 ……………. […]

Since the data came from ModuleA function parameter it was passed from the previous call:

0:009> kL 4 # ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 00 0381f844 00df3d1e 0x10101 01 0381f860 00ca0a15 ModuleA!Bar+0×3e 02 0381f888 00ca07d4 ModuleB!Foo2+0×75 03 0381f8e4 00ca0ae3 ModuleB!Foo1+0xb4

0:009> ub 00ca0a15 ModuleB!Foo2+0×61: 00ca0a01 add esp,10h 00ca0a04 and edi,0FFFFFFF8h 00ca0a07 lea edx,[esp+20h] 00ca0a0b push edx 00ca0a0c push ebp 00ca0a0d push edi 00ca0a0e push esi 00ca0a0f call dword ptr [ModuleA!_imp__Bar (00cc32c0)]

The passed parameter looks like the second argument of Bar and ModuleB!Foo2, and the first argument of ModuleB!Foo1:

0:009> dps 0381f860 0381f860 064bf018 0381f864 00ca0a15 ModuleB!Foo2+0x75 ; return address when calling ModuleA!Bar 0381f868 064bf018 0381f86c 0b925970 0381f870 0c86a949 0381f874 0381f898 0381f878 0b925970 0381f87c 09584580 0381f880 064bf018 0381f884 0381f8c4 0381f888 0b925970 0381f88c 00ca07d4 ModuleB!Foo1+0xb4 ; return address when calling ModuleB!Foo2 0381f890 064bf018 0381f894 0b925970 0381f898 09584580 0381f89c 064bf018 0381f8a0 00ccd7d0 0381f8a4 0381f954 0381f8a8 00000000 0381f8ac 00000001 0381f8b0 00000000 0381f8b4 0b926d18 0381f8b8 00000000 0381f8bc 00000005 0381f8c0 00000000 0381f8c4 0b925970 0381f8c8 00000000 0381f8cc 00000000 0381f8d0 00000000 0381f8d4 00000000 0381f8d8 00000000 0381f8dc 00000000 0381f8e0 00000000 0381f8e4 00000000 0381f8e8 00ca0ae3 ModuleB!iFoo0+0×13 ; return address when calling ModuleB!Foo1 0381f8ec 0b925970 0381f8f0 00000000

We see Parameter Flow analysis pattern:

0:009> kvL 4 # ChildEBP RetAddr Args to Child WARNING: Frame IP not in any known module. Following frames may be wrong. 00 0381f844 00df3d1e 064bf018 0b925970 06501dd8 0×10101 01 0381f860 00ca0a15 064bf018 0b925970 0c86a949 ModuleA!Bar+0×3e 02 0381f888 00ca07d4 064bf018 0b925970 09584580 ModuleB!Foo2+0×75 03 0381f8e4 00ca0ae3 0b925970 00000000 09584008 ModuleB!Foo1+0xb4

The example is from x86 Windows where parameters are passed through stack, and, therefore, are seen in the verbose stack trace output (kv). For x64 Windows systems such parameters may be Hidden and what we see in verbose output can be False Function Parameters. However, we can still track in some cases certain values found in Execution Residue and associate them with particular stack trace frames (including Past Stack Traces).

Parameter Flow memory analysis pattern is similar to trace analysis Data Flow pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 233)

November 18th, 2015

With the possibility of process cloning (reflection) starting from Windows 7 it is possible to get memory snapshots (Clone Dump) from a process clone (similar to fork API in Unix). Procdump tool has -r switch for that purpose. We checked this with x64 Windows 7 notepad.exe. We got two memory dumps: one is a clone with this stack trace:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dbgcfg.dmp] User Mini Dump File with Full Memory: Only application data is available

Comment: ' *** procdump -ma -r Notepad.exe *** Manual dump'

0:000> ~*k

. 0 Id: 25ec.147c Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Child-SP RetAddr Call Site 00 00000000`02c8fd38 00000000`7733aae7 ntdll!NtSuspendThread+0xa 01 00000000`02c8fd40 00000000`77165a4d ntdll!RtlpProcessReflectionStartup+0×2e7 02 00000000`02c8fe30 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd 03 00000000`02c8fe60 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The process memory has all address space of the original process including module list and heap structure:

0:000> lmn start end module name 00000000`77050000 00000000`7714a000 user32 user32.dll 00000000`77150000 00000000`77270000 kernel32 kernel32.dll 00000000`77270000 00000000`77419000 ntdll ntdll.dll 00000000`ff030000 00000000`ff065000 notepad notepad.exe 000007fe`f57d0000 000007fe`f5841000 winspool winspool.drv 000007fe`fb730000 000007fe`fb786000 uxtheme uxtheme.dll 000007fe`fb910000 000007fe`fbb04000 comctl32 comctl32.dll 000007fe`fbf00000 000007fe`fbf0c000 version version.dll 000007fe`fceb0000 000007fe`fcebf000 CRYPTBASE CRYPTBASE.dll 000007fe`fd310000 000007fe`fd37c000 KERNELBASE KERNELBASE.dll 000007fe`fd3d0000 000007fe`fd499000 usp10 usp10.dll 000007fe`fd4a0000 000007fe`fd511000 shlwapi shlwapi.dll 000007fe`fd520000 000007fe`fd64d000 rpcrt4 rpcrt4.dll 000007fe`fd650000 000007fe`fd66f000 sechost sechost.dll 000007fe`fd680000 000007fe`fd6ae000 imm32 imm32.dll 000007fe`fd6b0000 000007fe`fd78b000 advapi32 advapi32.dll 000007fe`fd810000 000007fe`fd8a7000 comdlg32 comdlg32.dll 000007fe`fdd60000 000007fe`fddff000 msvcrt msvcrt.dll 000007fe`fdfe0000 000007fe`fed69000 shell32 shell32.dll 000007fe`fed70000 000007fe`fedd7000 gdi32 gdi32.dll 000007fe`ff0b0000 000007fe`ff1b9000 msctf msctf.dll 000007fe`ff1c0000 000007fe`ff1ce000 lpk lpk.dll 000007fe`ff1d0000 000007fe`ff2a7000 oleaut32 oleaut32.dll 000007fe`ff2b0000 000007fe`ff349000 clbcatq clbcatq.dll 000007fe`ff350000 000007fe`ff553000 ole32 ole32.dll

0:000> !address -summary

Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 48 7ff`faa06000 ( 8.000 TB) 100.00% Image 129 0`01e79000 ( 30.473 MB) 35.47% 0.00% 21 0`01d10000 ( 29.063 MB) 33.83% 0.00% Other 9 0`016be000 ( 22.742 MB) 26.47% 0.00% Heap 26 0`00320000 ( 3.125 MB) 3.64% 0.00% Stack 3 0`00080000 ( 512.000 kB) 0.58% 0.00% TEB 1 0`00002000 ( 8.000 kB) 0.01% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_IMAGE 130 0`01e7a000 ( 30.477 MB) 35.47% 0.00% MEM_PRIVATE 47 0`01449000 ( 20.285 MB) 23.61% 0.00% MEM_MAPPED 11 0`00c97000 ( 12.590 MB) 14.65% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 50 7ff`fc096000 ( 8.000 TB) 100.00% MEM_COMMIT 176 0`02c10000 ( 44.063 MB) 51.29% 0.00% MEM_RESERVE 12 0`0134a000 ( 19.289 MB) 22.45% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READONLY 83 0`01b46000 ( 27.273 MB) 31.75% 0.00% PAGE_EXECUTE_READ 25 0`00f6d000 ( 15.426 MB) 17.95% 0.00% PAGE_WRITECOPY 48 0`00126000 ( 1.148 MB) 1.34% 0.00% PAGE_READWRITE 18 0`00032000 ( 200.000 kB) 0.23% 0.00% PAGE_READWRITE|PAGE_GUARD 2 0`00005000 ( 20.000 kB) 0.02% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 0`ff065000 7fd`f676b000 ( 7.992 TB) Image 7fe`fe4cb000 0`0089e000 ( 8.617 MB) 0`7f0e0000 0`00f00000 ( 15.000 MB) Other 0`00610000 0`01590000 ( 21.563 MB) Heap 0`003b8000 0`000c8000 ( 800.000 kB) Stack 0`02c10000 0`0006c000 ( 432.000 kB) TEB 7ff`fffdb000 0`00002000 ( 8.000 kB) PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

0:000> !heap -s

****************************************************** NT HEAP STATS BELOW ****************************************************** LFH Key : 0x000000381021167d Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ------------------------------------------------------------------------------------- 0000000000280000 00000002 1024 412 1024 14 4 1 0 0 LFH 0000000000250000 00001002 1088 256 1088 5 2 2 0 0 LFH 0000000001cc0000 00001002 64 8 64 3 1 1 0 0 0000000001e60000 00001002 512 120 512 49 3 1 0 0 0000000001dc0000 00001002 512 8 512 2 1 1 0 0 -------------------------------------------------------------------------------------

The other dump saved is a minidump from which we can get thread information for Execution Residue (raw stack data) and reconstruct stack traces in Clone Dump:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dmp] Comment: ' *** procdump -ma -r Notepad.exe *** Manual dump' User Mini Dump File: Only registers, stack and portions of memory are available

0:000> ~ . 0 Id: 87c.27f4 Suspend: 0 Teb: 000007ff`fffdd000 Unfrozen

0:000> k # Child-SP RetAddr Call Site 00 00000000`0016fac8 00000000`77069e9e 0x77069e6a 01 00000000`0016fad0 00000000`00000000 0x77069e9e

0:000> r rax=0000000000000000 rbx=000000000016fb40 rcx=0000000000280000 rdx=0000000000000000 rsi=0000000000000001 rdi=0000000000000000 rip=0000000077069e6a rsp=000000000016fac8 rbp=00000000ff030000 r8=000000000016f8e8 r9=00000000000a0cdc r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 00000000`77069e6a c3 ret

Now we can see the original stack trace in Clone Dump:

0:000> k =000000000016fac8 ff # Child-SP RetAddr Call Site 00 00000000`0016fac8 00000000`77069e9e ntdll!NtSuspendThread+0xa 01 00000000`0016fad0 00000000`ff031064 user32!GetMessageW+0x34 02 00000000`0016fb00 00000000`ff03133c notepad!WinMain+0x182 03 00000000`0016fb80 00000000`77165a4d notepad!DisplayNonGenuineDlgWorker+0x2da 04 00000000`0016fc40 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd 05 00000000`0016fc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Since we know TEB address from the minidump we can get stack region boundaries in Clone Dump:

0:000> dt _NT_TIB 000007fffffdd000 ntdll!_NT_TIB +0x000 ExceptionList : (null) +0x008 StackBase : 0x00000000`00170000 Void +0x010 StackLimit : 0x00000000`0015b000 Void +0x018 SubSystemTib : (null) +0x020 FiberData : 0x00000000`00001e00 Void +0x020 Version : 0x1e00 +0x028 ArbitraryUserPointer : (null) +0x030 Self : 0x000007ff`fffdd000 _NT_TIB

Now we can check Execution Residue (for example, for signs of Hidden Exceptions):

0:000> dpS 0x00000000`0015b000 0x00000000`00170000 [...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Trace Analysis Patterns (Part 119)

November 14th, 2015

Multiple traces and logs are usually collected for diagnosing distributed systems. Different tools and tracing settings (circular, sequential, file size limit) may be used, systems may be unsynchronized, and individual system tracing may be started at different times due to manual tracing setup and switching between systems. There may be Blackouts, Circular, and Truncated traces. When we analyze such a trace set (Inter-Correlation) we usually select one trace or log that is used as Calibrating Trace. It is used for measuring all other traces against Basic Facts such as start and end tracing times, and the time of the problem. One such scenario is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in CDF Analysis Tips and Tricks, Log Analysis, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Trace Analysis Patterns (Part 118)

November 10th, 2015

We would like to introduce Trace Dimension pattern to address the emerging complexity of logs from distributed environments. By a distributed environment we mean not only a collection of multiple computers (for example, client-server) but also terminal services environments with several different user sessions on one computer (OS) and even multiple user processes (IPC) in some cases. If some task can be performed on one machine or session or inside one process then splitting it across several computers, sessions, or processes usually results in logs with added Distributed Infrastructure Messages (DIMs) such as from proxies, channels and their management:

null

So, one of the trace simplification strategies is to request the reproduction and its tracing in a simplified environment (such as inside one terminal services session) to eliminate DIMs. In one case, we analyzed a trace for a clipboard paste problem in Windows terminal services environment. After a clipboard copy different data was pasted to different applications. The same behavior was observed for application processes running inside different sessions and processes running inside one session. However, the log was collected for the more complex multiple session scenario with many False Positive Errors which completely disappeared from one session scenario log.

DIM abbreviation played a role in naming this pattern. Additionally, if sessions can be considered a second dimension, then separate VMs can be considered as a third dimension, separate clouds as a 4th dimension, and so on.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in CDF Analysis Tips and Tricks, Log Analysis, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 232)

October 31st, 2015

We already introduced Active Thread pattern variant for Mac OS X. Here we provide an example for Windows. Unless we have Evental Dumps, Active Threads in Windows are usually threads from Busy System or Spiking Threads, and, therefore, represent an abnormal behavior since most threads are waiting or calling some API. For Evental Dumps they may be just normal threads:

0:000> r rax=0000000000000006 rbx=0000000000000003 rcx=0000000000000018 rdx=0000000000000000 rsi=000000000028c601 rdi=0000000002bee25e rip=000007feff1a5a09 rsp=000000000028c380 rbp=0000000000000000 r8=0000000000000000 r9=00000000001653a0 r10=000000000000000e r11=000000000000000a r12=0000000000000006 r13=000000000028ca38 r14=0000000002bec888 r15=0000000000173630 iopl=0 nv up ei pl nz ac po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216 usp10!otlChainingLookup::apply+0×2f9: 000007fe`ff1a5a09 498d0c06 lea rcx,[r14+rax]

0:000> k # Child-SP RetAddr Call Site 00 00000000`0028c380 000007fe`ff19f4f2 usp10!otlChainingLookup::apply+0×2f9 01 00000000`0028c4b0 000007fe`ff19e777 usp10!ApplyLookup+0×592 02 00000000`0028c5a0 000007fe`ff19a634 usp10!ApplyFeatures+0×777 03 00000000`0028c860 000007fe`ff181800 usp10!SubstituteOtlGlyphs+0×224 04 00000000`0028c910 000007fe`ff174cc0 usp10!GenericEngineGetGlyphs+0×1000 05 00000000`0028ccb0 000007fe`ff1389c5 usp10!ShlShape+0×7a0 06 00000000`0028ced0 000007fe`ff147363 usp10!ScriptShape+0×205 07 00000000`0028cf70 000007fe`ff148ac9 usp10!RenderItemNoFallback+0×433 08 00000000`0028d030 000007fe`ff148d86 usp10!RenderItemWithFallback+0×129 09 00000000`0028d080 000007fe`ff14a5f7 usp10!RenderItem+0×36 0a 00000000`0028d0d0 000007fe`ff13b2c9 usp10!ScriptStringAnalyzeGlyphs+0×277 0b 00000000`0028d170 000007fe`fdd616bf usp10!ScriptStringAnalyse+0×399 0c 00000000`0028d1f0 000007fe`fdd614cc lpk!LpkCharsetDraw+0×4eb 0d 00000000`0028d3b0 00000000`774e85c5 lpk!LpkDrawTextEx+0×68 0e 00000000`0028d420 00000000`774e865c user32!DT_DrawStr+0xa6 0f 00000000`0028d4c0 00000000`774e826c user32!DT_DrawJustifiedLine+0xa6 10 00000000`0028d530 00000000`774e6cc8 user32!DrawTextExWorker+0×442 11 00000000`0028d640 000007fe`fbd840d1 user32!DrawTextW+0×57 12 00000000`0028d6b0 000007fe`fbd83e49 comctl32!CLVView::_ComputeLabelSizeWorker+0×1d1 13 00000000`0028da40 000007fe`fbd8cc48 comctl32!CLVView::v_RecomputeLabelSize+0×1f9 14 00000000`0028dd70 000007fe`fbda9d24 comctl32!CLVListView::v_DrawItem+0×284 15 00000000`0028e110 000007fe`fbd9773b comctl32!CLVDrawItemManager::DrawItem+0×4c0 16 00000000`0028e170 000007fe`fbd95f8e comctl32!CLVDrawManager::_PaintItems+0×3df 17 00000000`0028e3b0 000007fe`fbd95e87 comctl32!CLVDrawManager::_PaintWorkArea+0xda 18 00000000`0028e430 000007fe`fbd95cff comctl32!CLVDrawManager::_OnPaintWorkAreas+0×147 19 00000000`0028e4c0 000007fe`fbd06f1b comctl32!CLVDrawManager::_OnPaint+0×14b 1a 00000000`0028e570 000007fe`fbd06011 comctl32!CListView::WndProc+0xebf 1b 00000000`0028e770 00000000`774e9bd1 comctl32!CListView::s_WndProc+0×6cd 1c 00000000`0028e7d0 00000000`774e3bfc user32!UserCallWinProcCheckWow+0×1ad 1d 00000000`0028e890 00000000`774e3b78 user32!CallWindowProcAorW+0xdc 1e 00000000`0028e8e0 000007fe`fbca6215 user32!CallWindowProcW+0×18 1f 00000000`0028e920 000007fe`fbca69a0 comctl32!CallOriginalWndProc+0×1d 20 00000000`0028e960 000007fe`fbca6768 comctl32!CallNextSubclassProc+0×8c 21 00000000`0028e9e0 000007fe`fde1096a comctl32!DefSubclassProc+0×7c 22 00000000`0028ea30 000007fe`fdde9df4 shell32!DefSubclassProc+0×56 23 00000000`0028ea60 000007fe`fbca69a0 shell32!CListViewHost::s_ListViewSubclassWndProc+0×267 24 00000000`0028eb40 000007fe`fbca6877 comctl32!CallNextSubclassProc+0×8c 25 00000000`0028ebc0 00000000`774e9bd1 comctl32!MasterSubclassProc+0xe7 26 00000000`0028ec60 00000000`774e72cb user32!UserCallWinProcCheckWow+0×1ad 27 00000000`0028ed20 00000000`774e6829 user32!DispatchClientMessage+0xc3 28 00000000`0028ed80 00000000`776211f5 user32!_fnDWORD+0×2d 29 00000000`0028ede0 00000000`774e6e5a ntdll!KiUserCallbackDispatcherContinue 2a 00000000`0028ee68 00000000`774e6e6c user32!NtUserDispatchMessage+0xa 2b 00000000`0028ee70 00000000`774e67c2 user32!DispatchMessageWorker+0×55b 2c 00000000`0028eef0 000007fe`fbcc34a4 user32!IsDialogMessageW+0×153 2d 00000000`0028ef80 000007fe`fbcc583f comctl32!Prop_IsDialogMessage+0×1f0 2e 00000000`0028eff0 000007fe`fbcc5c05 comctl32!_RealPropertySheet+0×31b 2f 00000000`0028f0c0 000007fe`ff214e68 comctl32!_PropertySheet+0×55 30 00000000`0028f100 000007fe`ff214bb1 comdlg32!Print_InvokePropertySheets+0×2c6 31 00000000`0028f660 000007fe`ff21499c comdlg32!PrintDlgExX+0×2be 32 00000000`0028f6c0 00000000`ffec250f comdlg32!PrintDlgExW+0×38 33 00000000`0028f730 00000000`ffec242b notepad!GetPrinterDCviaDialog+0xab 34 00000000`0028f7f0 00000000`ffec23e8 notepad!PrintIt+0×46 35 00000000`0028fb70 00000000`ffec14eb notepad!NPCommand+0xdb 36 00000000`0028fca0 00000000`774e9bd1 notepad!NPWndProc+0×540 37 00000000`0028fce0 00000000`774e98da user32!UserCallWinProcCheckWow+0×1ad 38 00000000`0028fda0 00000000`ffec10bc user32!DispatchMessageWorker+0×3b5 39 00000000`0028fe20 00000000`ffec133c notepad!WinMain+0×16f 3a 00000000`0028fea0 00000000`773c59ed notepad!DisplayNonGenuineDlgWorker+0×2da 3b 00000000`0028ff60 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 3c 00000000`0028ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see the thread is active, in the middle of the function. For comparison, the next 2 threads are waiting and calling API respectively:

0:000> ~1k # Child-SP RetAddr Call Site 00 00000000`02edf748 00000000`775eb037 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`02edf750 00000000`773c59ed ntdll!TppWaiterpThread+0×14d 02 00000000`02edf9f0 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 03 00000000`02edfa20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ~2k # Child-SP RetAddr Call Site 00 00000000`0344e048 000007fe`fd3f403e ntdll!NtUnmapViewOfSection+0xa 01 00000000`0344e050 00000000`774e2edf KERNELBASE!FreeLibrary+0xa4 02 00000000`0344e080 000007fe`fdddaab3 user32!PrivateExtractIconsW+0×34b 03 00000000`0344e5a0 000007fe`fdddac28 shell32!SHPrivateExtractIcons+0×50a 04 00000000`0344e870 000007fe`fdde34b4 shell32!SHDefExtractIconW+0×254 05 00000000`0344eb60 000007fe`fdde3435 shell32!CExtractIcon::_ExtractW+0xcd 06 00000000`0344ebe0 000007fe`fdf0d529 shell32!CExtractIconBase::Extract+0×21 07 00000000`0344ec20 000007fe`fdf0d2da shell32!IExtractIcon_Extract+0×43 08 00000000`0344ec60 000007fe`fdddfff0 shell32!_GetILIndexGivenPXIcon+0×22e 09 00000000`0344f100 000007fe`fdde27a4 shell32!_GetILIndexFromItem+0×87 0a 00000000`0344f1a0 000007fe`fddb6506 shell32!SHGetIconIndexFromPIDL+0×66 0b 00000000`0344f1d0 000007fe`fdedb186 shell32!MapIDListToIconILIndex+0×52 0c 00000000`0344f250 000007fe`fdddc54c shell32!CLoadSystemIconTask::InternalResumeRT+0×110 0d 00000000`0344f2e0 000007fe`fde0efcb shell32!CRunnableTask::Run+0xda 0e 00000000`0344f310 000007fe`fde12b56 shell32!CShellTask::TT_Run+0×124 0f 00000000`0344f340 000007fe`fde12cb2 shell32!CShellTaskThread::ThreadProc+0×1d2 10 00000000`0344f3e0 000007fe`ff2b3843 shell32!CShellTaskThread::s_ThreadProc+0×22 11 00000000`0344f410 00000000`775f15db shlwapi!ExecuteWorkItemThreadProc+0xf 12 00000000`0344f440 00000000`775f0c56 ntdll!RtlpTpWorkCallback+0×16b 13 00000000`0344f520 00000000`773c59ed ntdll!TppWorkerThread+0×5ff 14 00000000`0344f820 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 15 00000000`0344f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub ntdll!NtUnmapViewOfSection+0xa ntdll!NtAccessCheckAndAuditAlarm: 00000000`77621540 4c8bd1 mov r10,rcx 00000000`77621543 b826000000 mov eax,26h 00000000`77621548 0f05 syscall 00000000`7762154a c3 ret 00000000`7762154b 0f1f440000 nop dword ptr [rax+rax] ntdll!NtUnmapViewOfSection: 00000000`77621550 4c8bd1 mov r10,rcx 00000000`77621553 b827000000 mov eax,27h 00000000`77621558 0f05 syscall

Our Active Thread is not Spiking Thread since CPU consumption is minimal:

0:000> !runaway f User Mode Time Thread Time 0:1ca4 0 days 0:00:00.171 11:1fb0 0 days 0:00:00.000 10:f98 0 days 0:00:00.000 9:eb8 0 days 0:00:00.000 8:1b80 0 days 0:00:00.000 7:139c 0 days 0:00:00.000 6:1d9c 0 days 0:00:00.000 5:1b44 0 days 0:00:00.000 4:1edc 0 days 0:00:00.000 3:830 0 days 0:00:00.000 2:1638 0 days 0:00:00.000 1:1ab0 0 days 0:00:00.000 Kernel Mode Time Thread Time 0:1ca4 0 days 0:00:00.421 11:1fb0 0 days 0:00:00.000 10:f98 0 days 0:00:00.000 9:eb8 0 days 0:00:00.000 8:1b80 0 days 0:00:00.000 7:139c 0 days 0:00:00.000 6:1d9c 0 days 0:00:00.000 5:1b44 0 days 0:00:00.000 4:1edc 0 days 0:00:00.000 3:830 0 days 0:00:00.000 2:1638 0 days 0:00:00.000 1:1ab0 0 days 0:00:00.000 Elapsed Time Thread Time 11:1fb0 24692 days 13:29:46.335 0:1ca4 0 days 0:02:39.671 1:1ab0 0 days 0:01:48.239 2:1638 0 days 0:01:18.837 4:1edc 0 days 0:01:18.697 3:830 0 days 0:01:18.697 5:1b44 0 days 0:01:18.497 6:1d9c 0 days 0:01:18.387 7:139c 0 days 0:01:15.957 8:1b80 0 days 0:01:14.397 9:eb8 0 days 0:01:01.485 10:f98 0 days 0:00:39.849

However, the huge Elapsed Time for the thread #11 (most likely the value is uninitialized) and its stack trace suggest that the dump was saved on Create Thread debugging event by a debugger:

0:000> ~11k # Child-SP RetAddr Call Site 00 00000000`0666fb08 00000000`00000000 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 231)

October 26th, 2015

One of the customers of Software Diagnostics Services submitted memory dumps saved by DebugDiag to accompany software logs for the analysis of sudden process exit. We didn’t request such memory dumps and initially dismissed them. However, during software log analysis we decided to look at Adjoint Spaces to see whether there was some additional information in stack traces. We found out that those dumps were saved on each thread exit event. Since other threads were either waiting or Active Threads their analysis gave clues of process behavior before process exit. For example, we found ALPC Wait Chain to Coupled Process. The latter prompted us to analyze Coupled Activities in the software log and diagnose the possible problem there. Since saving memory dumps on thread creation and exit can be useful technique we decided to add Evental Dumps memory analysis pattern to our pattern catalog.

To illustrate this pattern we show Stack Trace Collection from notepad.exe. This process usually has just one thread. But, if we try to open a Print dialog the number of threads increases up to 12.

We attach WinDbg to notepad.exe process and set up debugging event filter (Debug \ Event Filters… menu) for Create Thread event with a command line as shown in this picture:

We then resume execution using g command and switch to Notepad. There we first open File \ Page Setup… dialog. We observe that a memory dump is saved. Then we open File \ Print… dialog and notice the creation 11 more process memory dump:

We now show stack traces from these dumps where we use ~*kc WinDbg command to minimize the amount of inessential for our purposes output:

// 1st dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!NtAlertThread 01 ntdll!TppWaiterEnqueueTransition 02 ntdll!TppWaitpSet 03 ntdll!TpSetWait 04 ntdll!TppTimerpInitTimerQueueQueue 05 ntdll!TppTimerpAllocTimerQueue 06 ntdll!TppTimerpAcquirePoolTimerQueue 07 ntdll!TppTimerAlloc 08 ntdll!TpAllocTimer 09 KERNELBASE!CreateThreadpoolTimer 0a rpcrt4!RPC_THREAD_POOL::CreateTimer 0b rpcrt4!GarbageCollectionNeeded 0c rpcrt4!LRPC_CASSOCIATION::RemoveReference 0d rpcrt4!LRPC_CCALL::`vector deleting destructor' 0e rpcrt4!LRPC_CCALL::FreeBuffer 0f rpcrt4!Ndr64pClientFinally 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 sechost!LsaLookupClose 13 sechost!LookupAccountNameInternal 14 sechost!LookupAccountNameLocalW 15 rpcrt4!RpcpLookupAccountNameDirect 16 rpcrt4!RpcpLookupAccountName 17 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 18 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 19 rpcrt4!RpcBindingSetAuthInfoExW 1a winspool!STRING_HANDLE_bind 1b rpcrt4!GenericHandleMgr 1c rpcrt4!ExplicitBindHandleMgr 1d rpcrt4!Ndr64pClientSetupTransferSyntax 1e rpcrt4!NdrpClientCall3 1f rpcrt4!NdrClientCall3 20 winspool!RpcSplOpenPrinter 21 winspool!OpenPrinterRPC 22 winspool!OpenPrinter2W 23 comdlg32!PrintOpenPrinter 24 comdlg32!PrintDlgX 25 comdlg32!PageSetupDlgX 26 comdlg32!PageSetupDlgW 27 notepad!NPCommand 28 notepad!NPWndProc 29 user32!UserCallWinProcCheckWow 2a user32!DispatchMessageWorker 2b notepad!WinMain 2c notepad!DisplayNonGenuineDlgWorker 2d kernel32!BaseThreadInitThunk 2e ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 2nd dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!ZwOpenKeyEx 01 kernel32!LocalBaseRegOpenKey 02 kernel32!RegOpenKeyExInternalW 03 kernel32!RegOpenKeyExW 04 rpcrt4!Server2003NegotiateDisable 05 rpcrt4!IsBindTimeFeatureNegotiationDisabled 06 rpcrt4!OSF_CCONNECTION::SendBindPacket 07 rpcrt4!OSF_CCONNECTION::ActuallyDoBinding 08 rpcrt4!OSF_CCONNECTION::OpenConnectionAndBind 09 rpcrt4!OSF_CCALL::BindToServer 0a rpcrt4!OSF_BINDING_HANDLE::InitCCallWithAssociation 0b rpcrt4!OSF_BINDING_HANDLE::AllocateCCall 0c rpcrt4!OSF_BINDING_HANDLE::NegotiateTransferSyntax 0d rpcrt4!I_RpcNegotiateTransferSyntax 0e rpcrt4!Ndr64pClientSetupTransferSyntax 0f rpcrt4!NdrpClientCall3 10 rpcrt4!NdrClientCall3 11 srvcli!NetShareEnum 12 ntshrui!CShareCache::RefreshNoCritSec 13 ntshrui!CShareCache::Refresh 14 ntshrui!DllMain 15 ntdll!RtlRunOnceExecuteOnce 16 kernel32!InitOnceExecuteOnce 17 ntshrui!DllGetClassObject 18 ole32!CClassCache::CDllPathEntry::DllGetClassObject 19 ole32!CClassCache::CDllFnPtrMoniker::BindToObjectNoSwitch 1a ole32!CClassCache::GetClassObject 1b ole32!CServerContextActivator::CreateInstance 1c ole32!ActivationPropertiesIn::DelegateCreateInstance 1d ole32!CApartmentActivator::CreateInstance 1e ole32!CProcessActivator::CCICallback 1f ole32!CProcessActivator::AttemptActivation 20 ole32!CProcessActivator::ActivateByContext 21 ole32!CProcessActivator::CreateInstance 22 ole32!ActivationPropertiesIn::DelegateCreateInstance 23 ole32!CClientContextActivator::CreateInstance 24 ole32!ActivationPropertiesIn::DelegateCreateInstance 25 ole32!ICoCreateInstanceEx 26 ole32!CoCreateInstance 27 shell32!_SHCoCreateInstance 28 shell32!SHExtCoCreateInstance 29 shell32!DCA_SHExtCoCreateInstance 2a shell32!CFSIconOverlayManager::_s_LoadIconOverlayIdentifiers 2b shell32!CFSIconOverlayManager::CreateInstance 2c shell32!IconOverlayManagerInit 2d shell32!GetIconOverlayManager 2e shell32!FileIconInit 2f shell32!Shell_GetImageLists 30 comdlg32!CPrintDialog::CPrintDialog 31 comdlg32!Print_GeneralDlgProc 32 user32!UserCallDlgProcCheckWow 33 user32!DefDlgProcWorker 34 user32!InternalCreateDialog 35 user32!CreateDialogIndirectParamAorW 36 user32!CreateDialogIndirectParamW 37 comctl32!_CreatePageDialog 38 comctl32!_CreatePage 39 comctl32!PageChange 3a comctl32!InitPropSheetDlg 3b comctl32!PropSheetDlgProc 3c user32!UserCallDlgProcCheckWow 3d user32!DefDlgProcWorker 3e user32!InternalCreateDialog 3f user32!CreateDialogIndirectParamAorW 40 user32!CreateDialogIndirectParamW 41 comctl32!_RealPropertySheet 42 comctl32!_PropertySheet 43 comdlg32!Print_InvokePropertySheets 44 comdlg32!PrintDlgExX 45 comdlg32!PrintDlgExW 46 notepad!GetPrinterDCviaDialog 47 notepad!PrintIt 48 notepad!NPCommand 49 notepad!NPWndProc 4a user32!UserCallWinProcCheckWow 4b user32!DispatchMessageWorker 4c notepad!WinMain 4d notepad!DisplayNonGenuineDlgWorker 4e kernel32!BaseThreadInitThunk 4f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 3rd dump: we see 2 threads start at the same time

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!RtlCompareMemoryUlong 01 ntdll!RtlpAllocateHeap 02 ntdll!RtlAllocateHeap 03 ntdll!RtlDebugAllocateHeap 04 ntdll! ?? ::FNODOBFM::`string' 05 ntdll!RtlAllocateHeap 06 ole32!CRpcResolver::GetThreadWinstaDesktop 07 ole32!CRpcResolver::GetConnection 08 ole32!CoInitializeSecurity 09 ole32!InitializeSecurity 0a ole32!ChannelProcessInitialize 0b ole32!CComApartment::InitRemoting 0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0d ole32!CGIPTable::RegisterInterfaceInGlobal 0e shell32!MarshalToGIT 0f shell32!CBrowserProgressAggregator::BeginSession 10 shell32!IUnknown_BeginBrowserProgressSession 11 shell32!CDefView::CreateViewWindow3 12 shell32!CExplorerBrowser::_CreateViewWindow 13 shell32!CExplorerBrowser::_SwitchView 14 shell32!CExplorerBrowser::_BrowseToView 15 shell32!CExplorerBrowser::_BrowseObjectInternal 16 shell32!CExplorerBrowser::_OnBrowseObject 17 shell32!CExplorerBrowser::BrowseObject 18 comdlg32!CPrintDialog::CreatePrintBrowser 19 comdlg32!CPrintDialog::OnInitDialog 1a comdlg32!Print_GeneralDlgProc 1b user32!UserCallDlgProcCheckWow 1c user32!DefDlgProcWorker 1d user32!InternalCreateDialog 1e user32!CreateDialogIndirectParamAorW 1f user32!CreateDialogIndirectParamW 20 comctl32!_CreatePageDialog 21 comctl32!_CreatePage 22 comctl32!PageChange 23 comctl32!InitPropSheetDlg 24 comctl32!PropSheetDlgProc 25 user32!UserCallDlgProcCheckWow 26 user32!DefDlgProcWorker 27 user32!InternalCreateDialog 28 user32!CreateDialogIndirectParamAorW 29 user32!CreateDialogIndirectParamW 2a comctl32!_RealPropertySheet 2b comctl32!_PropertySheet 2c comdlg32!Print_InvokePropertySheets 2d comdlg32!PrintDlgExX 2e comdlg32!PrintDlgExW 2f notepad!GetPrinterDCviaDialog 30 notepad!PrintIt 31 notepad!NPCommand 32 notepad!NPWndProc 33 user32!UserCallWinProcCheckWow 34 user32!DispatchMessageWorker 35 notepad!WinMain 36 notepad!DisplayNonGenuineDlgWorker 37 kernel32!BaseThreadInitThunk 38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupOpenLocalPolicy 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!RpcBindingSetAuthInfoExW 0c winspool!STRING_HANDLE_bind 0d rpcrt4!GenericHandleMgr 0e rpcrt4!ExplicitBindHandleMgr 0f rpcrt4!Ndr64pClientSetupTransferSyntax 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 winspool!RpcSplOpenPrinter 13 winspool!OpenPrinterRPC 14 winspool!OpenPrinter2W 15 prncache!PrintCache::Listeners::Listener::Start 16 prncache!PrintCache::Listeners::Listener::StartCB 17 ntdll!TppWorkpExecuteCallback 18 ntdll!TppWorkerThread 19 kernel32!BaseThreadInitThunk 1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 0 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 4th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!RtlCompareMemoryUlong 01 ntdll!RtlpAllocateHeap 02 ntdll!RtlAllocateHeap 03 ntdll!RtlDebugAllocateHeap 04 ntdll! ?? ::FNODOBFM::`string' 05 ntdll!RtlAllocateHeap 06 ole32!CRpcResolver::GetThreadWinstaDesktop 07 ole32!CRpcResolver::GetConnection 08 ole32!CoInitializeSecurity 09 ole32!InitializeSecurity 0a ole32!ChannelProcessInitialize 0b ole32!CComApartment::InitRemoting 0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0d ole32!CGIPTable::RegisterInterfaceInGlobal 0e shell32!MarshalToGIT 0f shell32!CBrowserProgressAggregator::BeginSession 10 shell32!IUnknown_BeginBrowserProgressSession 11 shell32!CDefView::CreateViewWindow3 12 shell32!CExplorerBrowser::_CreateViewWindow 13 shell32!CExplorerBrowser::_SwitchView 14 shell32!CExplorerBrowser::_BrowseToView 15 shell32!CExplorerBrowser::_BrowseObjectInternal 16 shell32!CExplorerBrowser::_OnBrowseObject 17 shell32!CExplorerBrowser::BrowseObject 18 comdlg32!CPrintDialog::CreatePrintBrowser 19 comdlg32!CPrintDialog::OnInitDialog 1a comdlg32!Print_GeneralDlgProc 1b user32!UserCallDlgProcCheckWow 1c user32!DefDlgProcWorker 1d user32!InternalCreateDialog 1e user32!CreateDialogIndirectParamAorW 1f user32!CreateDialogIndirectParamW 20 comctl32!_CreatePageDialog 21 comctl32!_CreatePage 22 comctl32!PageChange 23 comctl32!InitPropSheetDlg 24 comctl32!PropSheetDlgProc 25 user32!UserCallDlgProcCheckWow 26 user32!DefDlgProcWorker 27 user32!InternalCreateDialog 28 user32!CreateDialogIndirectParamAorW 29 user32!CreateDialogIndirectParamW 2a comctl32!_RealPropertySheet 2b comctl32!_PropertySheet 2c comdlg32!Print_InvokePropertySheets 2d comdlg32!PrintDlgExX 2e comdlg32!PrintDlgExW 2f notepad!GetPrinterDCviaDialog 30 notepad!PrintIt 31 notepad!NPCommand 32 notepad!NPWndProc 33 user32!UserCallWinProcCheckWow 34 user32!DispatchMessageWorker 35 notepad!WinMain 36 notepad!DisplayNonGenuineDlgWorker 37 kernel32!BaseThreadInitThunk 38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupOpenLocalPolicy 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!RpcBindingSetAuthInfoExW 0c winspool!STRING_HANDLE_bind 0d rpcrt4!GenericHandleMgr 0e rpcrt4!ExplicitBindHandleMgr 0f rpcrt4!Ndr64pClientSetupTransferSyntax 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 winspool!RpcSplOpenPrinter 13 winspool!OpenPrinterRPC 14 winspool!OpenPrinter2W 15 prncache!PrintCache::Listeners::Listener::Start 16 prncache!PrintCache::Listeners::Listener::StartCB 17 ntdll!TppWorkpExecuteCallback 18 ntdll!TppWorkerThread 19 kernel32!BaseThreadInitThunk 1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!LdrpInitializeThread 01 ntdll!LdrpInitialize 02 ntdll!LdrInitializeThunk

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 5th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupClose 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_FAST_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!LRPC_FAST_BINDING_HANDLE::LRPC_FAST_BINDING_HANDLE 0c rpcrt4!LrpcCreateFastBindingHandle 0d rpcrt4!RpcBindingCreateW 0e ole32!CFastBH::CreateFromBindingString 0f ole32!CFastBH::GetOrCreate 10 ole32!CRpcResolver::GetConnection 11 ole32!CoInitializeSecurity 12 ole32!InitializeSecurity 13 ole32!ChannelProcessInitialize 14 ole32!CComApartment::InitRemoting 15 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 16 ole32!CGIPTable::RegisterInterfaceInGlobal 17 shell32!MarshalToGIT 18 shell32!CBrowserProgressAggregator::BeginSession 19 shell32!IUnknown_BeginBrowserProgressSession 1a shell32!CDefView::CreateViewWindow3 1b shell32!CExplorerBrowser::_CreateViewWindow 1c shell32!CExplorerBrowser::_SwitchView 1d shell32!CExplorerBrowser::_BrowseToView 1e shell32!CExplorerBrowser::_BrowseObjectInternal 1f shell32!CExplorerBrowser::_OnBrowseObject 20 shell32!CExplorerBrowser::BrowseObject 21 comdlg32!CPrintDialog::CreatePrintBrowser 22 comdlg32!CPrintDialog::OnInitDialog 23 comdlg32!Print_GeneralDlgProc 24 user32!UserCallDlgProcCheckWow 25 user32!DefDlgProcWorker 26 user32!InternalCreateDialog 27 user32!CreateDialogIndirectParamAorW 28 user32!CreateDialogIndirectParamW 29 comctl32!_CreatePageDialog 2a comctl32!_CreatePage 2b comctl32!PageChange 2c comctl32!InitPropSheetDlg 2d comctl32!PropSheetDlgProc 2e user32!UserCallDlgProcCheckWow 2f user32!DefDlgProcWorker 30 user32!InternalCreateDialog 31 user32!CreateDialogIndirectParamAorW 32 user32!CreateDialogIndirectParamW 33 comctl32!_RealPropertySheet 34 comctl32!_PropertySheet 35 comdlg32!Print_InvokePropertySheets 36 comdlg32!PrintDlgExX 37 comdlg32!PrintDlgExW 38 notepad!GetPrinterDCviaDialog 39 notepad!PrintIt 3a notepad!NPCommand 3b notepad!NPWndProc 3c user32!UserCallWinProcCheckWow 3d user32!DispatchMessageWorker 3e notepad!WinMain 3f notepad!DisplayNonGenuineDlgWorker 40 kernel32!BaseThreadInitThunk 41 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcSplOpenPrinter 05 winspool!OpenPrinterRPC 06 winspool!OpenPrinter2W 07 prncache!PrintCache::Listeners::Listener::Start 08 prncache!PrintCache::Listeners::Listener::StartCB 09 ntdll!TppWorkpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcEnumPrinters 05 winspool!EnumPrintersW 06 prncache!PrintCache::Listeners::ConnectionListener::EnumConnectionsAndRegister 07 prncache!PrintCache::Listeners::ConnectionListener::UpdateCB 08 ntdll!TppWorkpExecuteCallback 09 ntdll!TppWorkerThread 0a kernel32!BaseThreadInitThunk 0b ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 6th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!NtAlpcConnectPort 01 rpcrt4!LRPC_CASSOCIATION::AlpcConnect 02 rpcrt4!LRPC_CASSOCIATION::Connect 03 rpcrt4!LRPC_BASE_BINDING_HANDLE::DriveStateForward 04 rpcrt4!LRPC_FAST_BINDING_HANDLE::Bind 05 rpcrt4!RpcBindingBind 06 ole32!CFastBH::CreateFromBindingString 07 ole32!CFastBH::GetOrCreate 08 ole32!CRpcResolver::GetConnection 09 ole32!CoInitializeSecurity 0a ole32!InitializeSecurity 0b ole32!ChannelProcessInitialize 0c ole32!CComApartment::InitRemoting 0d ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0e ole32!CGIPTable::RegisterInterfaceInGlobal 0f shell32!MarshalToGIT 10 shell32!CBrowserProgressAggregator::BeginSession 11 shell32!IUnknown_BeginBrowserProgressSession 12 shell32!CDefView::CreateViewWindow3 13 shell32!CExplorerBrowser::_CreateViewWindow 14 shell32!CExplorerBrowser::_SwitchView 15 shell32!CExplorerBrowser::_BrowseToView 16 shell32!CExplorerBrowser::_BrowseObjectInternal 17 shell32!CExplorerBrowser::_OnBrowseObject 18 shell32!CExplorerBrowser::BrowseObject 19 comdlg32!CPrintDialog::CreatePrintBrowser 1a comdlg32!CPrintDialog::OnInitDialog 1b comdlg32!Print_GeneralDlgProc 1c user32!UserCallDlgProcCheckWow 1d user32!DefDlgProcWorker 1e user32!InternalCreateDialog 1f user32!CreateDialogIndirectParamAorW 20 user32!CreateDialogIndirectParamW 21 comctl32!_CreatePageDialog 22 comctl32!_CreatePage 23 comctl32!PageChange 24 comctl32!InitPropSheetDlg 25 comctl32!PropSheetDlgProc 26 user32!UserCallDlgProcCheckWow 27 user32!DefDlgProcWorker 28 user32!InternalCreateDialog 29 user32!CreateDialogIndirectParamAorW 2a user32!CreateDialogIndirectParamW 2b comctl32!_RealPropertySheet 2c comctl32!_PropertySheet 2d comdlg32!Print_InvokePropertySheets 2e comdlg32!PrintDlgExX 2f comdlg32!PrintDlgExW 30 notepad!GetPrinterDCviaDialog 31 notepad!PrintIt 32 notepad!NPCommand 33 notepad!NPWndProc 34 user32!UserCallWinProcCheckWow 35 user32!DispatchMessageWorker 36 notepad!WinMain 37 notepad!DisplayNonGenuineDlgWorker 38 kernel32!BaseThreadInitThunk 39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!RtlEnterCriticalSection 01 ntdll!RtlDebugAllocateHeap 02 ntdll! ?? ::FNODOBFM::`string' 03 ntdll!RtlAllocateHeap 04 rpcrt4!AllocWrapper 05 rpcrt4!SID_CACHE::Query 06 rpcrt4!RpcpLookupAccountName 07 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 08 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 09 rpcrt4!RpcBindingSetAuthInfoExW 0a winspool!STRING_HANDLE_bind 0b rpcrt4!GenericHandleMgr 0c rpcrt4!ExplicitBindHandleMgr 0d rpcrt4!Ndr64pClientSetupTransferSyntax 0e rpcrt4!NdrpClientCall3 0f rpcrt4!NdrClientCall3 10 winspool!RpcSplOpenPrinter 11 winspool!OpenPrinterRPC 12 winspool!OpenPrinter2W 13 prncache!PrintCache::Listeners::Listener::Start 14 prncache!PrintCache::Listeners::Listener::StartCB 15 ntdll!TppWorkpExecuteCallback 16 ntdll!TppWorkerThread 17 kernel32!BaseThreadInitThunk 18 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 user32!NtUserAttachThreadInput 01 shell32!CWaitTask::s_WaitBeforeCursing 02 ntdll!RtlpTpWaitCallback 03 ntdll!TppWaitpExecuteCallback 04 ntdll!TppWorkerThread 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 7th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 rpcrt4!RpcStringFreeW 01 ole32!CFastBH::CreateFromBindingString 02 ole32!CFastBH::GetOrCreate 03 ole32!CRpcResolver::GetConnection 04 ole32!CoInitializeSecurity 05 ole32!InitializeSecurity 06 ole32!ChannelProcessInitialize 07 ole32!CComApartment::InitRemoting 08 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 09 ole32!CGIPTable::RegisterInterfaceInGlobal 0a shell32!MarshalToGIT 0b shell32!CBrowserProgressAggregator::BeginSession 0c shell32!IUnknown_BeginBrowserProgressSession 0d shell32!CDefView::CreateViewWindow3 0e shell32!CExplorerBrowser::_CreateViewWindow 0f shell32!CExplorerBrowser::_SwitchView 10 shell32!CExplorerBrowser::_BrowseToView 11 shell32!CExplorerBrowser::_BrowseObjectInternal 12 shell32!CExplorerBrowser::_OnBrowseObject 13 shell32!CExplorerBrowser::BrowseObject 14 comdlg32!CPrintDialog::CreatePrintBrowser 15 comdlg32!CPrintDialog::OnInitDialog 16 comdlg32!Print_GeneralDlgProc 17 user32!UserCallDlgProcCheckWow 18 user32!DefDlgProcWorker 19 user32!InternalCreateDialog 1a user32!CreateDialogIndirectParamAorW 1b user32!CreateDialogIndirectParamW 1c comctl32!_CreatePageDialog 1d comctl32!_CreatePage 1e comctl32!PageChange 1f comctl32!InitPropSheetDlg 20 comctl32!PropSheetDlgProc 21 user32!UserCallDlgProcCheckWow 22 user32!DefDlgProcWorker 23 user32!InternalCreateDialog 24 user32!CreateDialogIndirectParamAorW 25 user32!CreateDialogIndirectParamW 26 comctl32!_RealPropertySheet 27 comctl32!_PropertySheet 28 comdlg32!Print_InvokePropertySheets 29 comdlg32!PrintDlgExX 2a comdlg32!PrintDlgExW 2b notepad!GetPrinterDCviaDialog 2c notepad!PrintIt 2d notepad!NPCommand 2e notepad!NPWndProc 2f user32!UserCallWinProcCheckWow 30 user32!DispatchMessageWorker 31 notepad!WinMain 32 notepad!DisplayNonGenuineDlgWorker 33 kernel32!BaseThreadInitThunk 34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcSplOpenPrinter 05 winspool!OpenPrinterRPC 06 winspool!OpenPrinter2W 07 prncache!PrintCache::Listeners::Listener::Start 08 prncache!PrintCache::Listeners::Listener::StartCB 09 ntdll!TppWorkpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 8th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 user32!NtUserSetWindowLongPtr 01 user32!SetWindowLongPtr 02 ole32!OXIDEntry::StartServer 03 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 04 ole32!CGIPTable::RegisterInterfaceInGlobal 05 shell32!MarshalToGIT 06 shell32!CBrowserProgressAggregator::BeginSession 07 shell32!IUnknown_BeginBrowserProgressSession 08 shell32!CDefView::CreateViewWindow3 09 shell32!CExplorerBrowser::_CreateViewWindow 0a shell32!CExplorerBrowser::_SwitchView 0b shell32!CExplorerBrowser::_BrowseToView 0c shell32!CExplorerBrowser::_BrowseObjectInternal 0d shell32!CExplorerBrowser::_OnBrowseObject 0e shell32!CExplorerBrowser::BrowseObject 0f comdlg32!CPrintDialog::CreatePrintBrowser 10 comdlg32!CPrintDialog::OnInitDialog 11 comdlg32!Print_GeneralDlgProc 12 user32!UserCallDlgProcCheckWow 13 user32!DefDlgProcWorker 14 user32!InternalCreateDialog 15 user32!CreateDialogIndirectParamAorW 16 user32!CreateDialogIndirectParamW 17 comctl32!_CreatePageDialog 18 comctl32!_CreatePage 19 comctl32!PageChange 1a comctl32!InitPropSheetDlg 1b comctl32!PropSheetDlgProc 1c user32!UserCallDlgProcCheckWow 1d user32!DefDlgProcWorker 1e user32!InternalCreateDialog 1f user32!CreateDialogIndirectParamAorW 20 user32!CreateDialogIndirectParamW 21 comctl32!_RealPropertySheet 22 comctl32!_PropertySheet 23 comdlg32!Print_InvokePropertySheets 24 comdlg32!PrintDlgExX 25 comdlg32!PrintDlgExW 26 notepad!GetPrinterDCviaDialog 27 notepad!PrintIt 28 notepad!NPCommand 29 notepad!NPWndProc 2a user32!UserCallWinProcCheckWow 2b user32!DispatchMessageWorker 2c notepad!WinMain 2d notepad!DisplayNonGenuineDlgWorker 2e kernel32!BaseThreadInitThunk 2f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 9th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 user32!NtUserPeekMessage 01 user32!PeekMessageW 02 shell32!PeekMessageWithWakeMask 03 shell32!SHProcessMessagesUntilEventsEx 04 shell32!CDefView::_SetItemCollection 05 shell32!CDefView::_CreateNewCollection 06 shell32!CDefView::CreateViewWindow3 07 shell32!CExplorerBrowser::_CreateViewWindow 08 shell32!CExplorerBrowser::_SwitchView 09 shell32!CExplorerBrowser::_BrowseToView 0a shell32!CExplorerBrowser::_BrowseObjectInternal 0b shell32!CExplorerBrowser::_OnBrowseObject 0c shell32!CExplorerBrowser::BrowseObject 0d comdlg32!CPrintDialog::CreatePrintBrowser 0e comdlg32!CPrintDialog::OnInitDialog 0f comdlg32!Print_GeneralDlgProc 10 user32!UserCallDlgProcCheckWow 11 user32!DefDlgProcWorker 12 user32!InternalCreateDialog 13 user32!CreateDialogIndirectParamAorW 14 user32!CreateDialogIndirectParamW 15 comctl32!_CreatePageDialog 16 comctl32!_CreatePage 17 comctl32!PageChange 18 comctl32!InitPropSheetDlg 19 comctl32!PropSheetDlgProc 1a user32!UserCallDlgProcCheckWow 1b user32!DefDlgProcWorker 1c user32!InternalCreateDialog 1d user32!CreateDialogIndirectParamAorW 1e user32!CreateDialogIndirectParamW 1f comctl32!_RealPropertySheet 20 comctl32!_PropertySheet 21 comdlg32!Print_InvokePropertySheets 22 comdlg32!PrintDlgExX 23 comdlg32!PrintDlgExW 24 notepad!GetPrinterDCviaDialog 25 notepad!PrintIt 26 notepad!NPCommand 27 notepad!NPWndProc 28 user32!UserCallWinProcCheckWow 29 user32!DispatchMessageWorker 2a notepad!WinMain 2b notepad!DisplayNonGenuineDlgWorker 2c kernel32!BaseThreadInitThunk 2d ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 KERNELBASE!WaitForSingleObjectEx 02 shlwapi!CreateThreadWorker 03 shlwapi!SHCreateThread 04 shell32!CEnumThread::Run 05 shell32!CEnumTask::_StartEnumThread 06 shell32!CEnumTask::_IncrEnumFolder 07 shell32!CEnumTask::InternalResumeRT 08 shell32!CRunnableTask::Run 09 shell32!CShellTask::TT_Run 0a shell32!CShellTaskThread::ThreadProc 0b shell32!CShellTaskThread::s_ThreadProc 0c shlwapi!ExecuteWorkItemThreadProc 0d ntdll!RtlpTpWorkCallback 0e ntdll!TppWorkerThread 0f kernel32!BaseThreadInitThunk 10 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.1310 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 10th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 kernel32!TlsGetValue 01 usp10!UspFreeMem 02 usp10!ScriptApplyDigitSubstitution 03 lpk!LpkCharsetDraw 04 lpk!LpkDrawTextEx 05 user32!DT_GetExtentMinusPrefixes 06 user32!NeedsEndEllipsis 07 user32!AddEllipsisAndDrawLine 08 user32!DrawTextExWorker 09 user32!DrawTextW 0a comctl32!CLVView::_ComputeLabelSizeWorker 0b comctl32!CLVView::v_RecomputeLabelSize 0c comctl32!CLVListView::v_DrawItem 0d comctl32!CLVDrawItemManager::DrawItem 0e comctl32!CLVDrawManager::_PaintItems 0f comctl32!CLVDrawManager::_PaintWorkArea 10 comctl32!CLVDrawManager::_OnPaintWorkAreas 11 comctl32!CLVDrawManager::_OnPaint 12 comctl32!CListView::WndProc 13 comctl32!CListView::s_WndProc 14 user32!UserCallWinProcCheckWow 15 user32!CallWindowProcAorW 16 user32!CallWindowProcW 17 comctl32!CallOriginalWndProc 18 comctl32!CallNextSubclassProc 19 comctl32!DefSubclassProc 1a shell32!DefSubclassProc 1b shell32!CListViewHost::s_ListViewSubclassWndProc 1c comctl32!CallNextSubclassProc 1d comctl32!MasterSubclassProc 1e user32!UserCallWinProcCheckWow 1f user32!DispatchClientMessage 20 user32!_fnDWORD 21 ntdll!KiUserCallbackDispatcherContinue 22 user32!NtUserDispatchMessage 23 user32!DispatchMessageWorker 24 user32!IsDialogMessageW 25 comctl32!Prop_IsDialogMessage 26 comctl32!_RealPropertySheet 27 comctl32!_PropertySheet 28 comdlg32!Print_InvokePropertySheets 29 comdlg32!PrintDlgExX 2a comdlg32!PrintDlgExW 2b notepad!GetPrinterDCviaDialog 2c notepad!PrintIt 2d notepad!NPCommand 2e notepad!NPWndProc 2f user32!UserCallWinProcCheckWow 30 user32!DispatchMessageWorker 31 notepad!WinMain 32 notepad!DisplayNonGenuineDlgWorker 33 kernel32!BaseThreadInitThunk 34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtQueryKey 01 kernel32!BaseRegGetKeySemantics 02 kernel32!BaseRegGetUserAndMachineClass 03 kernel32!LocalBaseRegQueryValue 04 kernel32!RegQueryValueExW 05 shlwapi!SHRegQueryValueW 06 shlwapi!SHRegGetValueW 07 shlwapi!SHQueryValueExW 08 shell32!_GetServerInfo 09 shell32!_SHCoCreateInstance 0a shell32!CRegFolder::_CreateCachedRegFolder 0b shell32!CRegFolder::_BindToItem 0c shell32!CRegFolder::BindToObject 0d shell32!CRegFolder::_BindToItem 0e shell32!CRegFolder::BindToObject 0f shell32!SHBindToObject 10 shell32!CIconOverlayTask::InternalResumeRT 11 shell32!CRunnableTask::Run 12 shell32!CShellTask::TT_Run 13 shell32!CShellTaskThread::ThreadProc 14 shell32!CShellTaskThread::s_ThreadProc 15 shlwapi!ExecuteWorkItemThreadProc 16 ntdll!RtlpTpWorkCallback 17 ntdll!TppWorkerThread 18 kernel32!BaseThreadInitThunk 19 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 11th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 usp10!GenericEngineGetGlyphs 01 usp10!ShlShape 02 usp10!ScriptShape 03 usp10!RenderItemNoFallback 04 usp10!RenderItemWithFallback 05 usp10!RenderItem 06 usp10!ScriptStringAnalyzeGlyphs 07 usp10!ScriptStringAnalyse 08 lpk!LpkCharsetDraw 09 lpk!LpkDrawTextEx 0a user32!DT_GetExtentMinusPrefixes 0b user32!NeedsEndEllipsis 0c user32!AddEllipsisAndDrawLine 0d user32!DrawTextExWorker 0e user32!DrawTextW 0f comctl32!CLVView::_ComputeLabelSizeWorker 10 comctl32!CLVView::v_RecomputeLabelSize 11 comctl32!CLVListView::v_DrawItem 12 comctl32!CLVDrawItemManager::DrawItem 13 comctl32!CLVDrawManager::_PaintItems 14 comctl32!CLVDrawManager::_PaintWorkArea 15 comctl32!CLVDrawManager::_OnPaintWorkAreas 16 comctl32!CLVDrawManager::_OnPaint 17 comctl32!CListView::WndProc 18 comctl32!CListView::s_WndProc 19 user32!UserCallWinProcCheckWow 1a user32!CallWindowProcAorW 1b user32!CallWindowProcW 1c comctl32!CallOriginalWndProc 1d comctl32!CallNextSubclassProc 1e comctl32!DefSubclassProc 1f shell32!DefSubclassProc 20 shell32!CListViewHost::s_ListViewSubclassWndProc 21 comctl32!CallNextSubclassProc 22 comctl32!MasterSubclassProc 23 user32!UserCallWinProcCheckWow 24 user32!DispatchClientMessage 25 user32!_fnDWORD 26 ntdll!KiUserCallbackDispatcherContinue 27 user32!NtUserDispatchMessage 28 user32!DispatchMessageWorker 29 user32!IsDialogMessageW 2a comctl32!Prop_IsDialogMessage 2b comctl32!_RealPropertySheet 2c comctl32!_PropertySheet 2d comdlg32!Print_InvokePropertySheets 2e comdlg32!PrintDlgExX 2f comdlg32!PrintDlgExW 30 notepad!GetPrinterDCviaDialog 31 notepad!PrintIt 32 notepad!NPCommand 33 notepad!NPWndProc 34 user32!UserCallWinProcCheckWow 35 user32!DispatchMessageWorker 36 notepad!WinMain 37 notepad!DisplayNonGenuineDlgWorker 38 kernel32!BaseThreadInitThunk 39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 ntdll!RtlpWaitOnCriticalSection 02 ntdll!RtlEnterCriticalSection 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 ntdll!RtlpWaitOnCriticalSection 02 ntdll!RtlEnterCriticalSection 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 user32!NtUserGetIconInfo 01 user32!CopyIcoCur 02 user32!InternalCopyImage 03 user32!CopyImage 04 comctl32!CImageList::_ReplaceIcon 05 comctl32!CImageList::ReplaceIcon 06 comctl32!CSparseImageList::ReplaceIcon 07 shell32!CIconCache::AddToBackIconTable 08 shell32!AddToBackIconTable 09 shell32!SHAddIconsToCache 0a shell32!_GetILIndexGivenPXIcon 0b shell32!_GetILIndexFromItem 0c shell32!SHGetIconIndexFromPIDL 0d shell32!MapIDListToIconILIndex 0e shell32!CLoadSystemIconTask::InternalResumeRT 0f shell32!CRunnableTask::Run 10 shell32!CShellTask::TT_Run 11 shell32!CShellTaskThread::ThreadProc 12 shell32!CShellTaskThread::s_ThreadProc 13 shlwapi!ExecuteWorkItemThreadProc 14 ntdll!RtlpTpWorkCallback 15 ntdll!TppWorkerThread 16 kernel32!BaseThreadInitThunk 17 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 gdi32!NtUserSelectPalette 01 gdi32!SelectPalette 02 gdi32!SetDIBits 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcFindNextPrinterChangeNotification 05 winspool!FindNextPrinterChangeNotification 06 prncache!PrintCache::Listeners::Listener::ProcessWait 07 prncache!PrintCache::Listeners::Listener::ProcessWaitCB 08 ntdll!TppWaitpExecuteCallback 09 ntdll!TppWorkerThread 0a kernel32!BaseThreadInitThunk 0b ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 12th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 usp10!otlChainingLookup::apply 01 usp10!ApplyLookup 02 usp10!ApplyFeatures 03 usp10!SubstituteOtlGlyphs 04 usp10!GenericEngineGetGlyphs 05 usp10!ShlShape 06 usp10!ScriptShape 07 usp10!RenderItemNoFallback 08 usp10!RenderItemWithFallback 09 usp10!RenderItem 0a usp10!ScriptStringAnalyzeGlyphs 0b usp10!ScriptStringAnalyse 0c lpk!LpkCharsetDraw 0d lpk!LpkDrawTextEx 0e user32!DT_DrawStr 0f user32!DT_DrawJustifiedLine 10 user32!DrawTextExWorker 11 user32!DrawTextW 12 comctl32!CLVView::_ComputeLabelSizeWorker 13 comctl32!CLVView::v_RecomputeLabelSize 14 comctl32!CLVListView::v_DrawItem 15 comctl32!CLVDrawItemManager::DrawItem 16 comctl32!CLVDrawManager::_PaintItems 17 comctl32!CLVDrawManager::_PaintWorkArea 18 comctl32!CLVDrawManager::_OnPaintWorkAreas 19 comctl32!CLVDrawManager::_OnPaint 1a comctl32!CListView::WndProc 1b comctl32!CListView::s_WndProc 1c user32!UserCallWinProcCheckWow 1d user32!CallWindowProcAorW 1e user32!CallWindowProcW 1f comctl32!CallOriginalWndProc 20 comctl32!CallNextSubclassProc 21 comctl32!DefSubclassProc 22 shell32!DefSubclassProc 23 shell32!CListViewHost::s_ListViewSubclassWndProc 24 comctl32!CallNextSubclassProc 25 comctl32!MasterSubclassProc 26 user32!UserCallWinProcCheckWow 27 user32!DispatchClientMessage 28 user32!_fnDWORD 29 ntdll!KiUserCallbackDispatcherContinue 2a user32!NtUserDispatchMessage 2b user32!DispatchMessageWorker 2c user32!IsDialogMessageW 2d comctl32!Prop_IsDialogMessage 2e comctl32!_RealPropertySheet 2f comctl32!_PropertySheet 30 comdlg32!Print_InvokePropertySheets 31 comdlg32!PrintDlgExX 32 comdlg32!PrintDlgExW 33 notepad!GetPrinterDCviaDialog 34 notepad!PrintIt 35 notepad!NPCommand 36 notepad!NPWndProc 37 user32!UserCallWinProcCheckWow 38 user32!DispatchMessageWorker 39 notepad!WinMain 3a notepad!DisplayNonGenuineDlgWorker 3b kernel32!BaseThreadInitThunk 3c ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtUnmapViewOfSection 01 KERNELBASE!FreeLibrary 02 user32!PrivateExtractIconsW 03 shell32!SHPrivateExtractIcons 04 shell32!SHDefExtractIconW 05 shell32!CExtractIcon::_ExtractW 06 shell32!CExtractIconBase::Extract 07 shell32!IExtractIcon_Extract 08 shell32!_GetILIndexGivenPXIcon 09 shell32!_GetILIndexFromItem 0a shell32!SHGetIconIndexFromPIDL 0b shell32!MapIDListToIconILIndex 0c shell32!CLoadSystemIconTask::InternalResumeRT 0d shell32!CRunnableTask::Run 0e shell32!CShellTask::TT_Run 0f shell32!CShellTaskThread::ThreadProc 10 shell32!CShellTaskThread::s_ThreadProc 11 shlwapi!ExecuteWorkItemThreadProc 12 ntdll!RtlpTpWorkCallback 13 ntdll!TppWorkerThread 14 kernel32!BaseThreadInitThunk 15 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 gdi32!ZwGdiSetDIBitsToDeviceInternal 01 gdi32!SetDIBitsToDevice 02 gdi32!SetDIBits 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!ZwMapViewOfSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CASSOCIATION::AlpcSendWaitReceivePort 02 rpcrt4!LRPC_BASE_CCALL::DoSendReceive 03 rpcrt4!LRPC_BASE_CCALL::SendReceive 04 rpcrt4!NdrpClientCall2 05 rpcrt4!NdrClientCall2 06 ole32!CRpcResolver::BulkUpdateOIDs 07 ole32!CROIDTable::ClientBulkUpdateOIDWithPingServer 08 ole32!CROIDTable::WorkerThreadLoop 09 ole32!CRpcThread::WorkerLoop 0a ole32!CRpcThreadCache::RpcWorkerThreadEntry 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

11 Id: 1344.1fb0 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Software Trace Analysis, Trace Analysis Patterns, x64 Windows | No Comments »

Spiking Thread, Top Module, Module Hint, and Memory Fluctuation: pattern cooperation

October 24th, 2015

We noticed that after restarting Windows 7 system on one of our notebooks it becomes very sluggish. Task Manager showed 25% CPU usage in one of svchost.exe processes and very high usage of physical memory. So we immediately dumped it using procdump. The resulted process memory dump was almost 1.5 GB. Although the analysis case is very simple and straightforward we decided to publish to show the value of crash and hang dump analysis in understanding abnormal software behavior in “user” context.

When we open the memory dump and run !runaway WinDbg command we immediately recognize Spiking Thread pattern:

0:000> !runaway f User Mode Time Thread Time 38:14a0 0 days 0:04:59.911 36:a88 0 days 0:00:00.187 2:41c 0 days 0:00:00.187 39:19f8 0 days 0:00:00.046 34:fa8 0 days 0:00:00.046 21:12b4 0 days 0:00:00.031 43:8f0 0 days 0:00:00.015 42:1504 0 days 0:00:00.015 41:1a20 0 days 0:00:00.015 40:978 0 days 0:00:00.015 33:e0c 0 days 0:00:00.015 32:ff8 0 days 0:00:00.015 22:1304 0 days 0:00:00.015 19:f68 0 days 0:00:00.015 9:664 0 days 0:00:00.015 8:660 0 days 0:00:00.015 6:518 0 days 0:00:00.015 4:4a4 0 days 0:00:00.015 51:160c 0 days 0:00:00.000 50:1590 0 days 0:00:00.000 49:15d8 0 days 0:00:00.000 48:ac8 0 days 0:00:00.000 47:14d0 0 days 0:00:00.000 46:1bfc 0 days 0:00:00.000 45:18e8 0 days 0:00:00.000 44:1448 0 days 0:00:00.000 37:1910 0 days 0:00:00.000 35:558 0 days 0:00:00.000 31:14b8 0 days 0:00:00.000 30:14b4 0 days 0:00:00.000 29:14ac 0 days 0:00:00.000 28:13d0 0 days 0:00:00.000 27:13c8 0 days 0:00:00.000 26:13b4 0 days 0:00:00.000 25:13b0 0 days 0:00:00.000 24:13a8 0 days 0:00:00.000 23:1328 0 days 0:00:00.000 20:12b0 0 days 0:00:00.000 18:e20 0 days 0:00:00.000 17:e10 0 days 0:00:00.000 16:dd0 0 days 0:00:00.000 15:ce0 0 days 0:00:00.000 14:754 0 days 0:00:00.000 13:718 0 days 0:00:00.000 12:678 0 days 0:00:00.000 11:674 0 days 0:00:00.000 10:668 0 days 0:00:00.000 7:548 0 days 0:00:00.000 5:4ac 0 days 0:00:00.000 3:4a0 0 days 0:00:00.000 1:418 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Kernel Mode Time Thread Time 38:14a0 0 days 0:00:55.707 36:a88 0 days 0:00:01.778 2:41c 0 days 0:00:00.405 34:fa8 0 days 0:00:00.109 9:664 0 days 0:00:00.062 43:8f0 0 days 0:00:00.046 42:1504 0 days 0:00:00.046 21:12b4 0 days 0:00:00.046 32:ff8 0 days 0:00:00.031 22:1304 0 days 0:00:00.031 18:e20 0 days 0:00:00.031 39:19f8 0 days 0:00:00.015 19:f68 0 days 0:00:00.015 6:518 0 days 0:00:00.015 3:4a0 0 days 0:00:00.015 1:418 0 days 0:00:00.015 51:160c 0 days 0:00:00.000 50:1590 0 days 0:00:00.000 49:15d8 0 days 0:00:00.000 48:ac8 0 days 0:00:00.000 47:14d0 0 days 0:00:00.000 46:1bfc 0 days 0:00:00.000 45:18e8 0 days 0:00:00.000 44:1448 0 days 0:00:00.000 41:1a20 0 days 0:00:00.000 40:978 0 days 0:00:00.000 37:1910 0 days 0:00:00.000 35:558 0 days 0:00:00.000 33:e0c 0 days 0:00:00.000 31:14b8 0 days 0:00:00.000 30:14b4 0 days 0:00:00.000 29:14ac 0 days 0:00:00.000 28:13d0 0 days 0:00:00.000 27:13c8 0 days 0:00:00.000 26:13b4 0 days 0:00:00.000 25:13b0 0 days 0:00:00.000 24:13a8 0 days 0:00:00.000 23:1328 0 days 0:00:00.000 20:12b0 0 days 0:00:00.000 17:e10 0 days 0:00:00.000 16:dd0 0 days 0:00:00.000 15:ce0 0 days 0:00:00.000 14:754 0 days 0:00:00.000 13:718 0 days 0:00:00.000 12:678 0 days 0:00:00.000 11:674 0 days 0:00:00.000 10:668 0 days 0:00:00.000 8:660 0 days 0:00:00.000 7:548 0 days 0:00:00.000 5:4ac 0 days 0:00:00.000 4:4a4 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Elapsed Time Thread Time 0:410 0 days 0:10:24.550 2:41c 0 days 0:10:24.534 1:418 0 days 0:10:24.534 4:4a4 0 days 0:10:24.331 3:4a0 0 days 0:10:24.331 5:4ac 0 days 0:10:24.269 6:518 0 days 0:10:23.957 7:548 0 days 0:10:23.817 8:660 0 days 0:10:22.176 9:664 0 days 0:10:22.156 10:668 0 days 0:10:22.126 11:674 0 days 0:10:22.026 12:678 0 days 0:10:21.986 13:718 0 days 0:10:20.066 14:754 0 days 0:10:20.056 15:ce0 0 days 0:10:15.131 16:dd0 0 days 0:10:14.641 17:e10 0 days 0:10:14.551 18:e20 0 days 0:10:14.531 19:f68 0 days 0:10:13.611 21:12b4 0 days 0:10:10.647 20:12b0 0 days 0:10:10.647 22:1304 0 days 0:10:10.553 23:1328 0 days 0:10:10.381 24:13a8 0 days 0:10:09.024 26:13b4 0 days 0:10:08.931 25:13b0 0 days 0:10:08.931 28:13d0 0 days 0:10:08.899 27:13c8 0 days 0:10:08.899 31:14b8 0 days 0:10:07.932 30:14b4 0 days 0:10:07.932 29:14ac 0 days 0:10:07.932 32:ff8 0 days 0:08:11.785 33:e0c 0 days 0:08:11.644 34:fa8 0 days 0:08:06.750 35:558 0 days 0:08:05.765 36:a88 0 days 0:08:05.127 37:1910 0 days 0:08:02.608 38:14a0 0 days 0:07:19.276 42:1504 0 days 0:04:55.634 41:1a20 0 days 0:04:55.634 40:978 0 days 0:04:55.634 39:19f8 0 days 0:04:55.634 43:8f0 0 days 0:04:55.618 44:1448 0 days 0:04:42.634 46:1bfc 0 days 0:04:20.945 45:18e8 0 days 0:04:20.945 47:14d0 0 days 0:02:42.515 48:ac8 0 days 0:01:27.434 50:1590 0 days 0:00:04.917 49:15d8 0 days 0:00:04.917 51:160c 0 days 0:00:03.387

We see that most of the time the thread was consuming CPU in user mode and less in kernel mode.

We then examine that thread stack trace:

0:000> ~38k # Child-SP RetAddr Call Site 00 00000000`0414ddd8 00000000`7769cf66 ntdll!ZwQueryPerformanceCounter+0xa *** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll - 01 00000000`0414dde0 000007fe`ee7ee7aa ntdll!RtlQueryPerformanceFrequency+0x16 02 00000000`0414de10 000007fe`ee7ee53f wuaueng!DllInstall+0×153da 03 00000000`0414de40 000007fe`ee7e0d99 wuaueng!DllInstall+0×1516f 04 00000000`0414e190 000007fe`ee7df542 wuaueng!DllInstall+0×79c9 05 00000000`0414e1f0 000007fe`ee7df57c wuaueng!DllInstall+0×6172 06 00000000`0414e2b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 07 00000000`0414e370 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 08 00000000`0414e430 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 09 00000000`0414e4f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0a 00000000`0414e5b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0b 00000000`0414e670 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0c 00000000`0414e730 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0d 00000000`0414e7f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0e 00000000`0414e8b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0f 00000000`0414e970 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 10 00000000`0414ea30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 11 00000000`0414eaf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 12 00000000`0414ebb0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 13 00000000`0414ec70 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 14 00000000`0414ed30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 15 00000000`0414edf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 16 00000000`0414eeb0 000007fe`ee7df2d7 wuaueng!DllInstall+0×61ac 17 00000000`0414ef70 000007fe`ee7e9b4f wuaueng!DllInstall+0×5f07 18 00000000`0414f020 000007fe`ee7eb7e8 wuaueng!DllInstall+0×1077f 19 00000000`0414f370 000007fe`ee8010b2 wuaueng!DllInstall+0×12418 1a 00000000`0414f5b0 000007fe`ee7fe53e wuaueng!DllInstall+0×27ce2 1b 00000000`0414f610 000007fe`ee7fccac wuaueng!DllInstall+0×2516e 1c 00000000`0414f660 000007fe`ee7dec19 wuaueng!DllInstall+0×238dc 1d 00000000`0414f690 000007fe`ee7de30f wuaueng!DllInstall+0×5849 1e 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0×4f3f 1f 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd 20 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that Top Module is from Windows Update Agent:

0:000> lmv m wuaueng Browse full module list start end module name 000007fe`ee740000 000007fe`ee9a0000 wuaueng (export symbols) wuaueng.dll Loaded symbol image file: wuaueng.dll Image path: c:\Windows\System32\wuaueng.dll Image name: wuaueng.dll Browse all global symbols functions data Timestamp: Wed May 14 17:21:24 2014 (53739804) CheckSum: 00265DEA ImageSize: 00260000 File version: 7.6.7600.320 Product version: 7.6.7600.320 File flags: 8 (Mask 3F) Private File OS: 40004 NT Win32 File type: 2.0 Dll File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® Windows® Operating System InternalName: wuaueng.dll OriginalFilename: wuaueng.dll ProductVersion: 7.6.7600.320 FileVersion: 7.6.7600.320 (winmain_wtr_wsus3sp2(oobla).140514-0916) FileDescription: Windows Update Agent LegalCopyright: © Microsoft Corporation. All rights reserved.

We also examine process address space:

0:000> !address -summary

Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 507 7ff`9373b000 ( 7.998 TB) 99.98% Heap 574 0`3e2ad000 ( 994.676 MB) 57.27% 0.01% 9786 0`2803e000 ( 640.242 MB) 36.87% 0.01% Image 874 0`049a6000 ( 73.648 MB) 4.24% 0.00% Stack 156 0`01a00000 ( 26.000 MB) 1.50% 0.00% Other 13 0`001bb000 ( 1.730 MB) 0.10% 0.00% TEB 52 0`00068000 ( 416.000 kB) 0.02% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 834 0`5345e000 ( 1.301 GB) 76.72% 0.02% MEM_MAPPED 9747 0`14ab0000 ( 330.688 MB) 19.04% 0.00% MEM_IMAGE 875 0`049a7000 ( 73.652 MB) 4.24% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 507 7ff`9373b000 ( 7.998 TB) 99.98% MEM_COMMIT 11164 0`513d8000 ( 1.269 GB) 74.85% 0.02% MEM_RESERVE 292 0`1b4dd000 ( 436.863 MB) 25.15% 0.01%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READWRITE 5533 0`41d6c000 ( 1.029 GB) 60.66% 0.01% PAGE_WRITECOPY 4911 0`098b9000 ( 152.723 MB) 8.79% 0.00% PAGE_READONLY 504 0`03204000 ( 50.016 MB) 2.88% 0.00% PAGE_EXECUTE_READ 164 0`02b29000 ( 43.160 MB) 2.49% 0.00% PAGE_READWRITE|PAGE_GUARD 52 0`00086000 ( 536.000 kB) 0.03% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 0`ff84b000 7fd`eed65000 ( 7.992 TB) Heap 0`83eb0000 0`01c17000 ( 28.090 MB) 0`93070000 0`0ffbd000 ( 255.738 MB) Image 7fe`fe18a000 0`0089e000 ( 8.617 MB) Stack 0`07140000 0`0007b000 ( 492.000 kB) Other 0`00760000 0`00181000 ( 1.504 MB) TEB 7ff`fff0a000 0`00002000 ( 8.000 kB) PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

We see that process heap occupies almost 1 GB. Let’s see its statistics:

0:000> !heap -s

****************************** NT HEAP STATS BELOW ****************************** LFH Key : 0x000000a57ddeb5ed Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ------------------------------------------------------------------------------------- Virtual block: 0000000051eb0000 - 0000000051eb0000 (size 0000000000000000) Virtual block: 0000000052930000 - 0000000052930000 (size 0000000000000000) Virtual block: 0000000026ba0000 - 0000000026ba0000 (size 0000000000000000) Virtual block: 0000000053e50000 - 0000000053e50000 (size 0000000000000000) Virtual block: 0000000054490000 - 0000000054490000 (size 0000000000000000) Virtual block: 00000000547a0000 - 00000000547a0000 (size 0000000000000000) Virtual block: 000000003bda0000 - 000000003bda0000 (size 0000000000000000) Virtual block: 0000000056030000 - 0000000056030000 (size 0000000000000000) Virtual block: 00000000567c0000 - 00000000567c0000 (size 0000000000000000) Virtual block: 00000000572a0000 - 00000000572a0000 (size 0000000000000000) Virtual block: 0000000057870000 - 0000000057870000 (size 0000000000000000) Virtual block: 0000000045a10000 - 0000000045a10000 (size 0000000000000000) Virtual block: 0000000058fb0000 - 0000000058fb0000 (size 0000000000000000) Virtual block: 0000000045c10000 - 0000000045c10000 (size 0000000000000000) Virtual block: 00000000599c0000 - 00000000599c0000 (size 0000000000000000) Virtual block: 0000000059ff0000 - 0000000059ff0000 (size 0000000000000000) Virtual block: 000000005ae20000 - 000000005ae20000 (size 0000000000000000) Virtual block: 000000005c5d0000 - 000000005c5d0000 (size 0000000000000000) Virtual block: 0000000054b90000 - 0000000054b90000 (size 0000000000000000) Virtual block: 000000005d070000 - 000000005d070000 (size 0000000000000000) Virtual block: 000000006e370000 - 000000006e370000 (size 0000000000000000) Virtual block: 000000006f8e0000 - 000000006f8e0000 (size 0000000000000000) Virtual block: 000000006ed20000 - 000000006ed20000 (size 0000000000000000) Virtual block: 0000000070890000 - 0000000070890000 (size 0000000000000000) Virtual block: 000000005e370000 - 000000005e370000 (size 0000000000000000) Virtual block: 000000005f5a0000 - 000000005f5a0000 (size 0000000000000000) Virtual block: 000000005fa60000 - 000000005fa60000 (size 0000000000000000) Virtual block: 000000005ffe0000 - 000000005ffe0000 (size 0000000000000000) Virtual block: 0000000060770000 - 0000000060770000 (size 0000000000000000) Virtual block: 0000000060aa0000 - 0000000060aa0000 (size 0000000000000000) Virtual block: 0000000061810000 - 0000000061810000 (size 0000000000000000) Virtual block: 0000000061a30000 - 0000000061a30000 (size 0000000000000000) Virtual block: 0000000061f00000 - 0000000061f00000 (size 0000000000000000) Virtual block: 0000000064470000 - 0000000064470000 (size 0000000000000000) Virtual block: 0000000064c00000 - 0000000064c00000 (size 0000000000000000) Virtual block: 00000000656d0000 - 00000000656d0000 (size 0000000000000000) Virtual block: 00000000660f0000 - 00000000660f0000 (size 0000000000000000) Virtual block: 0000000066530000 - 0000000066530000 (size 0000000000000000) Virtual block: 00000000669d0000 - 00000000669d0000 (size 0000000000000000) Virtual block: 00000000676a0000 - 00000000676a0000 (size 0000000000000000) Virtual block: 0000000067a70000 - 0000000067a70000 (size 0000000000000000) Virtual block: 0000000068a20000 - 0000000068a20000 (size 0000000000000000) Virtual block: 0000000069f10000 - 0000000069f10000 (size 0000000000000000) Virtual block: 000000006a6c0000 - 000000006a6c0000 (size 0000000000000000) Virtual block: 000000006ad80000 - 000000006ad80000 (size 0000000000000000) Virtual block: 000000006b9a0000 - 000000006b9a0000 (size 0000000000000000) Virtual block: 000000006bb40000 - 000000006bb40000 (size 0000000000000000) Virtual block: 000000006c4f0000 - 000000006c4f0000 (size 0000000000000000) Virtual block: 000000006dc30000 - 000000006dc30000 (size 0000000000000000) Virtual block: 000000006de10000 - 000000006de10000 (size 0000000000000000) Virtual block: 000000006ef80000 - 000000006ef80000 (size 0000000000000000) Virtual block: 00000000728f0000 - 00000000728f0000 (size 0000000000000000) Virtual block: 0000000071270000 - 0000000071270000 (size 0000000000000000) Virtual block: 0000000074030000 - 0000000074030000 (size 0000000000000000) Virtual block: 00000000746c0000 - 00000000746c0000 (size 0000000000000000) Virtual block: 00000000749d0000 - 00000000749d0000 (size 0000000000000000) Virtual block: 0000000074e50000 - 0000000074e50000 (size 0000000000000000) Virtual block: 0000000071730000 - 0000000071730000 (size 0000000000000000) Virtual block: 0000000075c30000 - 0000000075c30000 (size 0000000000000000) Virtual block: 0000000075e00000 - 0000000075e00000 (size 0000000000000000) Virtual block: 0000000075fe0000 - 0000000075fe0000 (size 0000000000000000) Virtual block: 00000000761a0000 - 00000000761a0000 (size 0000000000000000) Virtual block: 0000000076460000 - 0000000076460000 (size 0000000000000000) Virtual block: 0000000076620000 - 0000000076620000 (size 0000000000000000) Virtual block: 0000000077850000 - 0000000077850000 (size 0000000000000000) Virtual block: 0000000076c60000 - 0000000076c60000 (size 0000000000000000) Virtual block: 0000000078ec0000 - 0000000078ec0000 (size 0000000000000000) Virtual block: 0000000062b30000 - 0000000062b30000 (size 0000000000000000) Virtual block: 000000007a520000 - 000000007a520000 (size 0000000000000000) Virtual block: 000000007ab50000 - 000000007ab50000 (size 0000000000000000) Virtual block: 00000000770e0000 - 00000000770e0000 (size 0000000000000000) Virtual block: 000000007af00000 - 000000007af00000 (size 0000000000000000) Virtual block: 000000007c6b0000 - 000000007c6b0000 (size 0000000000000000) Virtual block: 000000007de50000 - 000000007de50000 (size 0000000000000000) Virtual block: 000000007e2d0000 - 000000007e2d0000 (size 0000000000000000) Virtual block: 000000007fff0000 - 000000007fff0000 (size 0000000000000000) Virtual block: 00000000628c0000 - 00000000628c0000 (size 0000000000000000) Virtual block: 00000000809b0000 - 00000000809b0000 (size 0000000000000000) Virtual block: 0000000080d80000 - 0000000080d80000 (size 0000000000000000) Virtual block: 0000000081310000 - 0000000081310000 (size 0000000000000000) Virtual block: 00000000631a0000 - 00000000631a0000 (size 0000000000000000) Virtual block: 0000000062c70000 - 0000000062c70000 (size 0000000000000000) Virtual block: 0000000092ca0000 - 0000000092ca0000 (size 0000000000000000) Virtual block: 0000000003940000 - 0000000003940000 (size 0000000000000000) Virtual block: 0000000081cd0000 - 0000000081cd0000 (size 0000000000000000) Virtual block: 0000000082260000 - 0000000082260000 (size 0000000000000000) Virtual block: 0000000077200000 - 0000000077200000 (size 0000000000000000) Virtual block: 000000007ec90000 - 000000007ec90000 (size 0000000000000000) Virtual block: 000000007edb0000 - 000000007edb0000 (size 0000000000000000) Virtual block: 000000007eec0000 - 000000007eec0000 (size 0000000000000000) Virtual block: 00000000829f0000 - 00000000829f0000 (size 0000000000000000) Virtual block: 0000000003770000 - 0000000003770000 (size 0000000000000000) Virtual block: 0000000082fc0000 - 0000000082fc0000 (size 0000000000000000) Virtual block: 0000000083210000 - 0000000083210000 (size 0000000000000000) Virtual block: 0000000083320000 - 0000000083320000 (size 0000000000000000) Virtual block: 0000000083760000 - 0000000083760000 (size 0000000000000000) Virtual block: 0000000083870000 - 0000000083870000 (size 0000000000000000) Virtual block: 0000000083eb0000 - 0000000083eb0000 (size 0000000000000000) Virtual block: 0000000085ad0000 - 0000000085ad0000 (size 0000000000000000) Virtual block: 0000000086290000 - 0000000086290000 (size 0000000000000000) 0000000000340000 00000002 210688 174796 210688 60711 775 45 100 1 LFH External fragmentation 34 % (775 free blocks) 0000000000010000 00008000 64 4 64 1 1 1 0 0 00000000005c0000 00001002 3136 1256 3136 278 39 3 0 0 LFH External fragmentation 22 % (39 free blocks) 0000000000960000 00001002 512 8 512 2 1 1 0 0 0000000001520000 00001002 512 276 512 2 10 1 0 0 LFH 00000000013f0000 00001002 512 8 512 2 1 1 0 0 0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH 00000000010b0000 00001002 512 8 512 2 1 1 0 0 00000000019b0000 00001002 512 308 512 5 8 1 0 0 LFH 0000000001220000 00001002 512 380 512 63 21 1 0 0 LFH 00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH 0000000001900000 00001002 512 156 512 133 1 1 0 0 00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH 0000000002e00000 00001002 48256 16656 48256 9836 98 13 0 1 LFH External fragmentation 59 % (98 free blocks) 0000000002f10000 00001002 3584 3280 3584 1773 79 3 0 5 LFH External fragmentation 54 % (79 free blocks) 00000000035c0000 00001002 64 8 64 3 1 1 0 0 0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH 00000000038c0000 00001002 512 8 512 3 1 1 0 0 0000000003ad0000 00001002 512 8 512 3 1 1 0 0 0000000003d40000 00001002 512 8 512 3 1 1 0 0 00000000005b0000 00001002 64 8 64 3 1 1 0 0 0000000004280000 00011002 512 64 512 59 2 1 0 0 00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH 0000000004270000 00001002 1088 304 1088 21 5 2 0 0 LFH 0000000006650000 00000002 1088 80 1088 10 2 2 0 0 0000000007050000 00001002 512 8 512 3 1 1 0 0 0000000007cc0000 00001002 512 8 512 3 1 1 0 0 0000000000be0000 00001002 64 24 64 17 2 1 0 0 0000000006540000 00001002 1536 544 1536 259 4 2 0 0 LFH 0000000000b40000 00001002 64 8 64 3 1 1 0 0 0000000004950000 00001002 512 8 512 3 1 1 0 0 ————————————————————————————-

We see many large virtual blocks allocated with less than 200 KB of normal blocks committed. We suspect possible Memory Leak. Let’s check a few such virtual blocks:

0:000> !address 0000000062b30000

Usage: Heap Base Address: 00000000`62b30000 End Address: 00000000`62c65000 Region Size: 00000000`00135000 ( 1.207 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`62b30000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×62b30000

Content source: 1 (target), length: 135000

0:000> !address 0000000003770000

Usage: Heap Base Address: 00000000`03770000 End Address: 00000000`03870000 Region Size: 00000000`00100000 ( 1.000 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`03770000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×3770000

Content source: 1 (target), length: 100000

0:000> !address 0000000085ad0000

Usage: Heap Base Address: 00000000`85ad0000 End Address: 00000000`8628d000 Region Size: 00000000`007bd000 ( 7.738 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`85ad0000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×85ad0000

Content source: 1 (target), length: 7bd000

We see they all contain assembly packages for KB articles for various updates (Module Hints):

0:000> dc 00000000`85ad0000 00000000`85ad0000 86290000 00000000 83eb0000 00000000 ..)…………. 00000000`85ad0010 00000000 00000000 00000000 00000000 ……………. 00000000`85ad0020 007bd000 00000000 007bd000 00000000 ..{…….{….. 00000000`85ad0030 00000000 00000000 f333297f 04000000 ………)3….. 00000000`85ad0040 00000001 007bc51a 00000004 00000000 ……{……… 00000000`85ad0050 0061003c 00730073 006d0065 006c0062 <.a.s.s.e.m.b.l. 00000000`85ad0060 00200079 006d0078 006e006c 003d0073 y. .x.m.l.n.s.=. 00000000`85ad0070 00750022 006e0072 0073003a 00680063 “.u.r.n.:.s.c.h.

0:000> du 00000000`85ad0050 00000000`85ad0050 “<assembly xmlns=”urn:schemas-mic” 00000000`85ad0090 “rosoft-com:asm.v3″ manifestVersi” 00000000`85ad00d0 “on=”1.0″ description=”Fix for KB” 00000000`85ad0110 “2798162″ displayName=”default” c” 00000000`85ad0150 “ompany=”Microsoft Corporation” c” 00000000`85ad0190 “opyright=”Microsoft Corporation”" 00000000`85ad01d0 ” supportInformation=”http://supp” 00000000`85ad0210 “ort.microsoft.com/?kbid=2798162″” 00000000`85ad0250 ” creationTimeStamp=”2013-04-15T0″ 00000000`85ad0290 “4:10:39Z” lastUpdateTimeStamp=”2″ 00000000`85ad02d0 “013-04-15T04:10:39Z”><assemblyid” 00000000`85ad0310 “entity name=”Package_for_KB27981″

However, in less than 20 minutes CPU and memory consumption normalized and the new saved process memory dump was less than 150 KB. We open it and see that CPU consumption happened for about 15 minutes (in the past) with the consuming thread now #32 instead of #38:

0:000> !runaway f User Mode Time Thread Time 32:14a0 0 days 0:13:24.029 33:b6c 0 days 0:00:00.078 41:174c 0 days 0:00:00.062 40:1bd8 0 days 0:00:00.062 30:fa8 0 days 0:00:00.046 45:650 0 days 0:00:00.015 42:4ac 0 days 0:00:00.015 39:990 0 days 0:00:00.015 29:e0c 0 days 0:00:00.015 28:ff8 0 days 0:00:00.015 18:1304 0 days 0:00:00.015 17:f68 0 days 0:00:00.015 7:664 0 days 0:00:00.015 6:660 0 days 0:00:00.015 4:518 0 days 0:00:00.015 3:4a4 0 days 0:00:00.015 47:9c8 0 days 0:00:00.000 46:f20 0 days 0:00:00.000 44:1440 0 days 0:00:00.000 43:11d0 0 days 0:00:00.000 38:db4 0 days 0:00:00.000 37:6ac 0 days 0:00:00.000 36:c4c 0 days 0:00:00.000 35:ff4 0 days 0:00:00.000 34:1950 0 days 0:00:00.000 31:1910 0 days 0:00:00.000 27:14b8 0 days 0:00:00.000 26:14b4 0 days 0:00:00.000 25:14ac 0 days 0:00:00.000 24:13d0 0 days 0:00:00.000 23:13c8 0 days 0:00:00.000 22:13b4 0 days 0:00:00.000 21:13b0 0 days 0:00:00.000 20:13a8 0 days 0:00:00.000 19:1328 0 days 0:00:00.000 16:e20 0 days 0:00:00.000 15:e10 0 days 0:00:00.000 14:dd0 0 days 0:00:00.000 13:ce0 0 days 0:00:00.000 12:754 0 days 0:00:00.000 11:718 0 days 0:00:00.000 10:678 0 days 0:00:00.000 9:674 0 days 0:00:00.000 8:668 0 days 0:00:00.000 5:548 0 days 0:00:00.000 2:4a0 0 days 0:00:00.000 1:418 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Kernel Mode Time Thread Time 32:14a0 0 days 0:02:24.878 33:b6c 0 days 0:00:00.171 30:fa8 0 days 0:00:00.109 40:1bd8 0 days 0:00:00.093 43:11d0 0 days 0:00:00.078 42:4ac 0 days 0:00:00.062 34:1950 0 days 0:00:00.062 7:664 0 days 0:00:00.062 41:174c 0 days 0:00:00.046 28:ff8 0 days 0:00:00.031 18:1304 0 days 0:00:00.031 16:e20 0 days 0:00:00.031 1:418 0 days 0:00:00.031 45:650 0 days 0:00:00.015 17:f68 0 days 0:00:00.015 6:660 0 days 0:00:00.015 4:518 0 days 0:00:00.015 2:4a0 0 days 0:00:00.015 47:9c8 0 days 0:00:00.000 46:f20 0 days 0:00:00.000 44:1440 0 days 0:00:00.000 39:990 0 days 0:00:00.000 38:db4 0 days 0:00:00.000 37:6ac 0 days 0:00:00.000 36:c4c 0 days 0:00:00.000 35:ff4 0 days 0:00:00.000 31:1910 0 days 0:00:00.000 29:e0c 0 days 0:00:00.000 27:14b8 0 days 0:00:00.000 26:14b4 0 days 0:00:00.000 25:14ac 0 days 0:00:00.000 24:13d0 0 days 0:00:00.000 23:13c8 0 days 0:00:00.000 22:13b4 0 days 0:00:00.000 21:13b0 0 days 0:00:00.000 20:13a8 0 days 0:00:00.000 19:1328 0 days 0:00:00.000 15:e10 0 days 0:00:00.000 14:dd0 0 days 0:00:00.000 13:ce0 0 days 0:00:00.000 12:754 0 days 0:00:00.000 11:718 0 days 0:00:00.000 10:678 0 days 0:00:00.000 9:674 0 days 0:00:00.000 8:668 0 days 0:00:00.000 5:548 0 days 0:00:00.000 3:4a4 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Elapsed Time Thread Time 0:410 0 days 0:35:17.550 1:418 0 days 0:35:17.534 3:4a4 0 days 0:35:17.331 2:4a0 0 days 0:35:17.331 4:518 0 days 0:35:16.957 5:548 0 days 0:35:16.817 6:660 0 days 0:35:15.176 7:664 0 days 0:35:15.156 8:668 0 days 0:35:15.126 9:674 0 days 0:35:15.026 10:678 0 days 0:35:14.986 11:718 0 days 0:35:13.066 12:754 0 days 0:35:13.056 13:ce0 0 days 0:35:08.131 14:dd0 0 days 0:35:07.641 15:e10 0 days 0:35:07.551 16:e20 0 days 0:35:07.531 17:f68 0 days 0:35:06.611 18:1304 0 days 0:35:03.553 19:1328 0 days 0:35:03.381 20:13a8 0 days 0:35:02.024 22:13b4 0 days 0:35:01.931 21:13b0 0 days 0:35:01.931 24:13d0 0 days 0:35:01.899 23:13c8 0 days 0:35:01.899 27:14b8 0 days 0:35:00.932 26:14b4 0 days 0:35:00.932 25:14ac 0 days 0:35:00.932 28:ff8 0 days 0:33:04.785 29:e0c 0 days 0:33:04.644 30:fa8 0 days 0:32:59.750 31:1910 0 days 0:32:55.608 32:14a0 0 days 0:32:12.276 34:1950 0 days 0:18:39.607 33:b6c 0 days 0:18:39.607 35:ff4 0 days 0:17:43.530 36:c4c 0 days 0:08:45.458 38:db4 0 days 0:07:41.551 37:6ac 0 days 0:07:41.551 39:990 0 days 0:06:54.877 40:1bd8 0 days 0:06:54.867 41:174c 0 days 0:05:38.282 42:4ac 0 days 0:03:54.627 43:11d0 0 days 0:03:53.122 44:1440 0 days 0:03:51.627 45:650 0 days 0:02:15.536 47:9c8 0 days 0:00:11.100 46:f20 0 days 0:00:11.100

0:000> ~32k # Child-SP RetAddr Call Site 00 00000000`0414f558 000007fe`fd4f1430 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`0414f560 00000000`775706e0 KERNELBASE!WaitForMultipleObjectsEx+0xe8 *** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll - 02 00000000`0414f660 000007fe`ee7de250 kernel32!WaitForMultipleObjects+0xb0 03 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0x4e80 04 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd 05 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Process heap doesn’t have any large allocated virtual block with only almost 100 KB committed normal blocks:

0:000> !heap -s

**** NT HEAP STATS BELOW ****************************** LFH Key Termination on corruption Heap Flags (k) (k) (k) (k) length ------------------------- 0000000000340000 External fragmentation 0000000000010000 00008000 00000000005c0000 00001002 External fragmentation 0000000000960000 00001002 512 0000000001520000 00001002 00000000013f0000 00001002 512 0000000001790000 00001002 00000000010b0000 00001002 512 00000000019b0000 00001002 0000000001220000 00001002 00000000021e0000 00001002 0000000001900000 00001002 00000000024b0000 00001002 0000000002e00000 00001002 External fragmentation 0000000002f10000 00001002 External fragmentation 00000000035c0000 00001002 0000000003200000 00001002 00000000038c0000 00001002 512 0000000003ad0000 00001002 512 0000000003d40000 00001002 512 00000000005b0000 00001002 0000000004280000 00011002 00000000040c0000 00001002 0000000004270000 00001002 0000000007050000 00001002 512 0000000007cc0000 00001002 512 0000000000be0000 00001002 Virtual block: 0000000009b90000 0000000006540000 00001002 External fragmentation 0000000000b40000 00001002 0000000004950000 00001002 ———R ************************** : 0x000000a57ddeb5ed : ENABLED Reserv Commit Virt Free List UCR Virt Lock Fast blocks cont. heap ------------------------------------------------------------ 00000002 210688 94444 210688 85646 752 86 0 1 LFH 90 % (752 free blocks) 64 4 64 1 1 1 0 0 3136 1256 3136 282 39 3 0 0 LFH 22 % (39 free blocks) 8 512 2 1 1 0 0 512 280 512 3 10 1 0 0 LFH 8 512 2 1 1 0 0 512 248 512 0 6 1 0 0 LFH 8 512 2 1 1 0 0 512 320 512 6 12 1 0 0 LFH 512 380 512 63 22 1 0 0 LFH 512 256 512 11 3 1 0 0 LFH 512 156 512 133 1 1 0 0 1536 792 1536 27 4 2 0 0 LFH 48256 16656 48256 9870 101 13 0 1 LFH 59 % (101 free blocks) 3584 3280 3584 1805 73 3 0 5 LFH 55 % (73 free blocks) 64 8 64 3 1 1 0 0 512 388 512 2 3 1 0 0 LFH 8 512 3 1 1 0 0 8 512 3 1 1 0 0 8 512 3 1 1 0 0 64 8 64 3 1 1 0 0 512 64 512 59 2 1 0 0 1088 296 1088 14 6 2 0 0 LFH 1088 304 1088 22 6 2 0 0 LFH 8 512 3 1 1 0 0 8 512 3 1 1 0 0 64 24 64 17 2 1 0 0 - 0000000009b90000 (size 0000000000000000) 1536 1064 1536 775 11 2 1 0 LFH 72 % (11 free blocks) 64 8 64 3 1 1 0 0 512 312 512 55 12 1 0 0 LFH 12;————————————————————————-

So we consider the dump set as an instance of Memory Fluctuation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Trace Analysis Patterns (Part 117)

October 22nd, 2015

Typical software trace may contain several Error Messages with different error codes and different exception names with Exception Stack Traces. Searching for individual codes or exceptions in problem databases may show many matches. Searching for all of them may show nothing. Therefore, we can construct the set of all subsets from the set of codes and exceptions (a power set) and perform analytic reasoning (and a search) based on certain subsets based on the problem description, Trace Viewpoints such as Use Case Trails, Motifs, Focus of Tracing, Foreground Components, (Adjoint) Threads of Activity, and simply some Activity Regions and Message Sets.

The following picture illustrates Error Powerset analysis pattern with a trace that has 4 error messages where 2 messages have the same error code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Log Analysis, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Trace Analysis Patterns (Part 116)

October 18th, 2015

Sometimes we need to know about client-server interaction between components, threads, or processes in order to find out where the problem started. For example, if we have Error Message or Discontinuity in one PID Adjoint Thread of Activity, and we know that that process uses API from another PID, we can look at the latter PID Adjoint Thread to see if there are any Error Messages or other problems. The failure in the server can propagate to the client as illustrated in the following diagram:

We call this pattern Coupled Activities similar to Coupled Processes memory analysis pattern. It can help in Intra- and Inter-Correlation analysis, for example in choosing adjoint threads from Sheaf of Activities.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Patterns, Software Trace Analysis, Trace Analysis Patterns, Tracing and Design Patterns | No Comments »

Crash Dump Analysis Patterns (Part 230)

October 14th, 2015

Sometimes, a stack trace from Stack Trace Collection may look well-formed at the first sight like having an expected start frames, for example:

0:000> ~*k

[...]

# 19 Id: 16a4.21f4 Suspend: 0 Teb: 7e95b000 Unfrozen ChildEBP RetAddr 0c2de6b0 74eb112f ntdll!NtWaitForMultipleObjects+0xc 0c2de83c 76ca7b89 KERNELBASE!WaitForMultipleObjectsEx+0xcc 0c2de858 76d007bf kernel32!WaitForMultipleObjects+0x19 0c2dec98 76d00295 kernel32!WerpReportFaultInternal+0x50b 0c2deca8 76ce1709 kernel32!WerpReportFault+0x74 0c2decb0 74f5f705 kernel32!BasepReportFault+0x19 0c2ded3c 76fb4f84 KERNELBASE!UnhandledExceptionFilter+0x1f4 0c2ded54 76fb5728 ntdll!TppExceptionFilter+0x30 0c2ded64 76f5c95a ntdll!TppWorkerpInnerExceptionFilter+0xe 0c2df914 76ca7c04 ntdll!TppWorkerThread+0×87f5a 0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0×24 0c2df970 76f1acea ntdll!__RtlUserThreadStart+0×2f 0c2df980 00000000 ntdll!_RtlUserThreadStart+0×1b

[...]

So, we may think something wrong happened in ntdll!TppWorkerThread code (although 0×87f5a offset looks suspicious). However, in reality, in this case due to exception filter logic (or some other reason in different cases) we have Hidden Stack Trace. When looking at UnhandledExceptionFilter parameters (or raw stack as in the case of Hidden Exceptions) we find an exception context:

0:019> kv ChildEBP RetAddr Args to Child 0c2de6b0 74eb112f 00000003 0c2de880 00000001 ntdll!NtWaitForMultipleObjects+0xc 0c2de83c 76ca7b89 00000003 0c2de880 00000000 KERNELBASE!WaitForMultipleObjectsEx+0xcc 0c2de858 76d007bf 00000003 0c2de880 00000000 kernel32!WaitForMultipleObjects+0x19 0c2dec98 76d00295 00000000 00000001 00000000 kernel32!WerpReportFaultInternal+0x50b 0c2deca8 76ce1709 0c2ded3c 74f5f705 0c2ded94 kernel32!WerpReportFault+0x74 0c2decb0 74f5f705 0c2ded94 00000001 a79b7895 kernel32!BasepReportFault+0x19 0c2ded3c 76fb4f84 0c2ded94 0c2ded94 00000000 KERNELBASE!UnhandledExceptionFilter+0×1f4 0c2ded54 76fb5728 00000000 00000000 0c2df914 ntdll!TppExceptionFilter+0×30 0c2ded64 76f5c95a 0c2df8d0 76f00a70 0c2df914 ntdll!TppWorkerpInnerExceptionFilter+0xe 0c2df914 76ca7c04 0f79e380 76ca7be0 a5b45024 ntdll!TppWorkerThread+0×87f5a 0c2df928 76f1ad1f 0f79e380 a59141d7 00000000 kernel32!BaseThreadInitThunk+0×24 0c2df970 76f1acea ffffffff 76f00233 00000000 ntdll!__RtlUserThreadStart+0×2f 0c2df980 00000000 76ed4a00 0f79e380 00000000 ntdll!_RtlUserThreadStart+0×1b

0:019> dd 0c2ded94 L2 0c2ded94 0c2deef8 0c2def48

0:019> .cxr 0c2def48 eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10 eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 ntdll!RtlEnterCriticalSection+0×12: 76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

0:019> k *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr 0c2df3ac 7407999c ntdll!RtlEnterCriticalSection+0x12 0c2df3cc 7407acd6 ModuleA!DoWork+0x1b [...] 0c2df73c 76ee3aa7 ModuleA!ThreadPoolWorkCallback+0xa9 0c2df77c 76ee1291 ntdll!TppWorkpExecuteCallback+0x137 0c2df914 76ca7c04 ntdll!TppWorkerThread+0x48e 0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0x24 0c2df970 76f1acea ntdll!__RtlUserThreadStart+0x2f 0c2df980 00000000 ntdll!_RtlUserThreadStart+0x1b

We consider this a different pattern than Hidden Call because an entire stack (sub)trace is missing between UnhandledExceptionFilter and thread start frames:

ntdll!NtWaitForMultipleObjects KERNELBASE!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects kernel32!WerpReportFaultInternal kernel32!WerpReportFault kernel32!BasepReportFault KERNELBASE!UnhandledExceptionFilter [...] ntdll!TppWorkerThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

This pattern is also different from Past Stack Trace pattern because Hidden Stack Trace belongs to PRESENT time zone. Our example is also different from Hidden Exception analysis pattern and its recovered stack trace because exception processing is not hidden and shows Exception Stack Trace albeit with a hidden part.

We were also fortunate to have Stored Exception (accessible by !analyze -v command):

0:019> .exr -1 ExceptionAddress: 76f00fb2 (ntdll!RtlEnterCriticalSection+0x00000012) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000001 Parameter[1]: 15f237e5 Attempt to write to address 15f237e5

0:019> .ecxr eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10 eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 ntdll!RtlEnterCriticalSection+0x12: 76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

So this Hidden Stack Trace is detected straightforwardly. But in other cases, such as when we have Multiple Exceptions in a process dump or Stack Trace Collection from a complete memory dump, we need to pay attention to such a possibility.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Trace Analysis Patterns (Part 115)

October 12th, 2015

By analogy with Adjoint Thread of Activity we introduce Adjoint Message analysis pattern. Most if not all analysis patterns focus on log message text and consider TID, PID, Module, source file and function as its attributes. However, we can choose one of attributes and consider it as a message in its own right with the original message text consigned now as another attribute. Then we can analyze the structure of the trace from the perspective of that newly selected message:

Since the number of different message values now is smaller (for example, module names) compared to normal trace messages we can use them in protein-like encoding and structure analysis schemes (see Software Trace and Logs as Proteins). We metaphorically name Adjoint Messages as Amino-acid-Messages (A-Messages). We can also compress same message sequences into one message which may be useful for pattern matching (and even use different color intensities to represent message cardinalities):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Software Trace Analysis, Software Trace Analysis and Proteomics, Software Trace Reading, Software Trace Visualization, Trace Analysis Patterns | No Comments »

Trace Analysis Patterns (Part 114)

September 30th, 2015

Sometimes we have Periodic Message Blocks of a few adjacent messages, for example, when flags are translated into separate messages per bit. Then we may have a pattern of Sequence Repeat Anomaly when one of several message blocks have missing or added messages compared to the more numerous number of expected identical message blocks. Then Missing Message Message Context may be explored further. The following diagram illustrates the pattern:

The name of the pattern comes from the notion of repeated DNA sequences.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in CDF Analysis Tips and Tricks, Log Analysis, Software Trace Analysis, Software Trace Analysis and Genetics, Software Trace Analysis and Genomics, Software Trace Reading, Trace Analysis Patterns | No Comments »

Workaround Patterns (Part 5)

September 26th, 2015

We resume our workaround patterns for common reusable solutions to common software execution problems. In the past we proposed a general pattern of Axed Code for removing problem software behavior. The Shadow File blog post about fixing free() crashes by introducing heap metadata header with a data length set to zero led me to generalize and introduce a complementary Axed Data pattern. Such a pattern suggests cutting the data size specified in metadata memory plane (which may be separate from the data plane). In some cases, it may avoid buffer overwrites including Local and Shared in addition to Dynamic Memory Corruption (process heap and kernel pool). The following picture illustrates this general pattern approach.

Sometimes, it may even be possible to provide a workaround by cutting real file data, for example, by changing its file or database record size. But this is also done conceptually by changing file or database system metadata.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Workaround Patterns | No Comments »

Crash Dump Analysis Patterns (Part 229)

September 13th, 2015

The advent of virtual machines, the possibility of saving complete memory snapshots without interruption, and the ability to quickly convert such snapshots into a debugger readable memory dump format such as in the case of VMware allows to study how Stack Trace Collections and Wait Chains change over time in complex problem scenarios. Such Stack Trace Surface may also show service restarts if PID changes for processes of interest. We call this pattern by analogy with a memory dump surface where each line corresponds to an individual memory snapshot with coordinates from 0 to the highest address:

In case of orbifold memory space we have a case of a 3D volume (we may call 3D orbifold).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Mathematics of Debugging, Memory Visualization, Software Trace Analysis | No Comments »

Trace Analysis Patterns (Part 113)

September 12th, 2015

Recently we analyzed a few logs which ended with a specialized Activity Region from a subsystem that sets operational parameters. The problem description stated that the system became unresponsive after changing parameters in a certain sequence. Usually, for that system, when we stop logging (even after setting parameters) we end up with messages from some Background Components since some time passes between the end of setting parameters activity and the time the operator sends stop logging request:

However, in the problem case we see message flow stops right in the middle of parameter setting activity:

So we advised to check for any crashes or hangs, and, indeed, it was found that the system was actually experiencing system crashes, and we got memory dumps for analysis where we found Top Module from a 3rd-party vendor related to parameter setting activity.

Please also note an analogy here between normal thread stack traces from threads that are waiting most of the time and Spiking Thread stack trace caught up in the middle of some function.

We call this pattern Ruptured Trace after a ruptured computation.

Note, that if it is possible to restart the system and resume the same tracing we may get an instance of Blackout analysis pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Log Analysis, Software Trace Analysis, Software Trace Analysis Tips and Tricks, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 35b)

September 12th, 2015

Sometimes we notice the anomalies in object distribution in heaps and pools. Memory consumption may be high in case of big objects. Such anomalies may point to possible memory, handle, and object leaks. But it may also be a temporary condition (Memory Fluctuation) due to the large amount of queued or postponed work that can be solved by proper software configuration. Diagnosed anomalies may also direction troubleshooting efforts if they cluster around certain component(s) or specific functionality. The distribution can be assessed by both the total memory consumption and the total number of objects of a particular class.

Here’s an example of Object Distribution Anomaly analysis pattern from .NET heap. The output of !DumpHeap -stat WinDbg SOS extension command shows the abnormal distribution of objects related to SQL data queries:

Count TotalSize […] 2342 281040 System.Reflection.RuntimeParameterInfo 3868 309440 System.Data.Metadata.Edm.TypeUsage 13218 317232 System.Object 3484 390208 System.Reflection.RuntimeMethodInfo 6092 508044 System.Int32[] 2756 617344 System.Data.SqlClient._SqlMetaData 2770 822870 System.Char[] 24560 1375360 System.RuntimeType 18 4195296 System.Int64[] 449691 10792584 System.Data.SqlClient.SqlGen.SqlBuilder 449961 10799064 System.Int64 449691 14390112 System.Data.Query.InternalTrees.ComparisonOp 449695 14390240 System.Data.Query.InternalTrees.ConditionalOp 6360 15509435 System.Byte[] 449690 17987600 System.Data.Query.InternalTrees.ConstantOp 449938 17997520 System.Data.Query.InternalTrees.VarRefOp 450898 21643104 System.Data.Common.CommandTrees.DbPropertyExpression […]

The anomalous character of the distribution is also illustrated in the following diagrams:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 27e)

August 24th, 2015

This is another variant of Stack Trace Collection pattern that shows stack traces from threads currently execution on all CPUs. Although we can see the non-idle running threads from the stack traces corresponding to all processes and their threads we may also want to see idle thread stack traces too. Also, the corresponding WinDbg command (!running -t -i) is faster if we want to double check the output of !analyze -v command in case of BSOD. The latter command may show the stack trace from the current CPU instead of the stack trace from the thread running on a different CPU that caused a bugcheck. Here’s an example from one of the memory dumps for which !analyze -v command shows an incorrect stack trace in the output when we open the dump file. It reports the stack trace from CPU 0 but the bugcheck happened on CPU 1:

0: kd> !running -t -i

System Processors: (00000000000000ff) Idle Processors: (00000000000000fd)

Prcbs Current (pri) Next (pri) Idle 0 fffff801e5d85180 fffff801e5ddea00 ( 0) fffff801e5ddea00 ................

Child-SP RetAddr Call Site fffff801`e8c9eb60 fffff801`e5c69b74 hal!KeQueryPerformanceCounter+0x75 fffff801`e8c9eba0 fffff801`e5c69e01 nt!KiCheckStall+0x2c fffff801`e8c9ebd0 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 fffff801`e8c9ece0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b fffff801`e8c9ed30 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 fffff801`e8c9ee70 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 fffff801`e8c8c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf fffff801`e8c8c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa fffff801`e8c8c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 fffff801`e8c8cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 fffff801`e8c8cc60 00000000`00000000 nt!KiIdleLoop+0x2c

1 ffffd000f0975180 ffffe0000d726880 (12) ffffd000f09813c0 …………….

Child-SP RetAddr Call Site ffffd000`202cb618 fffff801`e5a1cc3c hal!HalpAcpiPmRegisterReadPort+0x1b ffffd000`202cb620 fffff801`e5a417e7 hal!HalpAcpiPmRegisterRead+0x30 ffffd000`202cb650 fffff801`e5c66af5 hal!HaliHaltSystem+0x53 ffffd000`202cb690 fffff801`e5c66741 nt!KiBugCheckDebugBreak+0×99 ffffd000`202cb6f0 fffff801`e5bd2aa4 nt!KeBugCheck2+0xc6d ffffd000`202cbe00 fffff801`e5bde4e9 nt!KeBugCheckEx+0×104 ffffd000`202cbe40 fffff801`e5bdcd3a nt!KiBugCheckDispatch+0×69 ffffd000`202cbf80 fffff800`913601da nt!KiPageFault+0×23a ffffd000`202cc118 fffff800`91363710 DriverA!memcpy+0×21a […]

2 ffffd000f09ee180 ffffd000f09fa3c0 ( 0) ffffd000f09fa3c0 ................

Child-SP RetAddr Call Site ffffd000`f09f9f88 fffff801`e5c69e01 nt!KiCheckStall+0xa ffffd000`f09f9f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`f09fa0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`f09fa0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`f09fa230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb5938e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb5938f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb593920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb593b10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb593c60 00000000`00000000 nt!KiIdleLoop+0x2c

3 ffffd000eb5e5180 ffffd000eb5f13c0 ( 0) ffffd000eb5f13c0 ................

Child-SP RetAddr Call Site ffffd000`eb5f0f60 fffff801`e5c69e01 nt!KiCheckStall+0x5f ffffd000`eb5f0f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb5f10a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb5f10f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb5f1230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb5fa8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb5fa8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb5fa920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb5fab10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb5fac60 00000000`00000000 nt!KiIdleLoop+0x2c

4 ffffd000f08d1180 ffffd000f08dd3c0 ( 0) ffffd000f08dd3c0 ................

Child-SP RetAddr Call Site ffffd000`f08dcf90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x227 ffffd000`f08dd0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`f08dd0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`f08dd230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb85b8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb85b8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb85b920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb85bb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb85bc60 00000000`00000000 nt!KiIdleLoop+0x2c

5 ffffd000eb8ad180 ffffd000eb8b93c0 ( 0) ffffd000eb8b93c0 ................

Child-SP RetAddr Call Site ffffd000`eb8b8f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb8b8f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb8b90a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb8b90f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb8b9230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb8db8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb8db8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb8db920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb8dbb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb8dbc60 00000000`00000000 nt!KiIdleLoop+0x2c

6 ffffd000eb92a180 ffffd000eb9363c0 ( 0) ffffd000eb9363c0 ................

Child-SP RetAddr Call Site ffffd000`eb935f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb935f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb9360a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb9360f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb936230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb93f8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb93f8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb93f920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb93fb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb93fc60 00000000`00000000 nt!KiIdleLoop+0x2c

7 ffffd000eb967180 ffffd000eb9733c0 ( 0) ffffd000eb9733c0 ................

Child-SP RetAddr Call Site ffffd000`eb972f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb972f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb9730a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb9730f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb973230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb97c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb97c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb97c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb97cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb97cc60 00000000`00000000 nt!KiIdleLoop+0x2c

This command is obviously faster than repeatedly switching to subsequent CPUs using ~s command and then checking the corresponding stack trace (k). It also helps in diagnosing Spiking Threads in kernel and complete memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Kernel Memory Dump Analysis, x64 Windows | 1 Comment »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta