Archive for October, 2015

Crash Dump Analysis Patterns (Part 232)

Saturday, October 31st, 2015

We already introduced Active Thread pattern variant for Mac OS X. Here we provide an example for Windows. Unless we have Evental Dumps, Active Threads in Windows are usually threads from Busy System or Spiking Threads, and, therefore, represent an abnormal behavior since most threads are waiting or calling some API. For Evental Dumps they may be just normal threads:

0:000> r
rax=0000000000000006 rbx=0000000000000003 rcx=0000000000000018
rdx=0000000000000000 rsi=000000000028c601 rdi=0000000002bee25e
rip=000007feff1a5a09 rsp=000000000028c380 rbp=0000000000000000
r8=0000000000000000 r9=00000000001653a0 r10=000000000000000e
r11=000000000000000a r12=0000000000000006 r13=000000000028ca38
r14=0000000002bec888 r15=0000000000173630
iopl=0 nv up ei pl nz ac po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216
usp10!otlChainingLookup::apply+0×2f9:
000007fe`ff1a5a09 498d0c06 lea rcx,[r14+rax]

0:000> k
# Child-SP RetAddr Call Site
00 00000000`0028c380 000007fe`ff19f4f2 usp10!otlChainingLookup::apply+0×2f9
01 00000000`0028c4b0 000007fe`ff19e777 usp10!ApplyLookup+0×592
02 00000000`0028c5a0 000007fe`ff19a634 usp10!ApplyFeatures+0×777
03 00000000`0028c860 000007fe`ff181800 usp10!SubstituteOtlGlyphs+0×224
04 00000000`0028c910 000007fe`ff174cc0 usp10!GenericEngineGetGlyphs+0×1000
05 00000000`0028ccb0 000007fe`ff1389c5 usp10!ShlShape+0×7a0
06 00000000`0028ced0 000007fe`ff147363 usp10!ScriptShape+0×205
07 00000000`0028cf70 000007fe`ff148ac9 usp10!RenderItemNoFallback+0×433
08 00000000`0028d030 000007fe`ff148d86 usp10!RenderItemWithFallback+0×129
09 00000000`0028d080 000007fe`ff14a5f7 usp10!RenderItem+0×36
0a 00000000`0028d0d0 000007fe`ff13b2c9 usp10!ScriptStringAnalyzeGlyphs+0×277
0b 00000000`0028d170 000007fe`fdd616bf usp10!ScriptStringAnalyse+0×399
0c 00000000`0028d1f0 000007fe`fdd614cc lpk!LpkCharsetDraw+0×4eb
0d 00000000`0028d3b0 00000000`774e85c5 lpk!LpkDrawTextEx+0×68
0e 00000000`0028d420 00000000`774e865c user32!DT_DrawStr+0xa6
0f 00000000`0028d4c0 00000000`774e826c user32!DT_DrawJustifiedLine+0xa6
10 00000000`0028d530 00000000`774e6cc8 user32!DrawTextExWorker+0×442
11 00000000`0028d640 000007fe`fbd840d1 user32!DrawTextW+0×57
12 00000000`0028d6b0 000007fe`fbd83e49 comctl32!CLVView::_ComputeLabelSizeWorker+0×1d1
13 00000000`0028da40 000007fe`fbd8cc48 comctl32!CLVView::v_RecomputeLabelSize+0×1f9
14 00000000`0028dd70 000007fe`fbda9d24 comctl32!CLVListView::v_DrawItem+0×284
15 00000000`0028e110 000007fe`fbd9773b comctl32!CLVDrawItemManager::DrawItem+0×4c0
16 00000000`0028e170 000007fe`fbd95f8e comctl32!CLVDrawManager::_PaintItems+0×3df
17 00000000`0028e3b0 000007fe`fbd95e87 comctl32!CLVDrawManager::_PaintWorkArea+0xda
18 00000000`0028e430 000007fe`fbd95cff comctl32!CLVDrawManager::_OnPaintWorkAreas+0×147
19 00000000`0028e4c0 000007fe`fbd06f1b comctl32!CLVDrawManager::_OnPaint+0×14b
1a 00000000`0028e570 000007fe`fbd06011 comctl32!CListView::WndProc+0xebf
1b 00000000`0028e770 00000000`774e9bd1 comctl32!CListView::s_WndProc+0×6cd
1c 00000000`0028e7d0 00000000`774e3bfc user32!UserCallWinProcCheckWow+0×1ad
1d 00000000`0028e890 00000000`774e3b78 user32!CallWindowProcAorW+0xdc
1e 00000000`0028e8e0 000007fe`fbca6215 user32!CallWindowProcW+0×18
1f 00000000`0028e920 000007fe`fbca69a0 comctl32!CallOriginalWndProc+0×1d
20 00000000`0028e960 000007fe`fbca6768 comctl32!CallNextSubclassProc+0×8c
21 00000000`0028e9e0 000007fe`fde1096a comctl32!DefSubclassProc+0×7c
22 00000000`0028ea30 000007fe`fdde9df4 shell32!DefSubclassProc+0×56
23 00000000`0028ea60 000007fe`fbca69a0 shell32!CListViewHost::s_ListViewSubclassWndProc+0×267
24 00000000`0028eb40 000007fe`fbca6877 comctl32!CallNextSubclassProc+0×8c
25 00000000`0028ebc0 00000000`774e9bd1 comctl32!MasterSubclassProc+0xe7
26 00000000`0028ec60 00000000`774e72cb user32!UserCallWinProcCheckWow+0×1ad
27 00000000`0028ed20 00000000`774e6829 user32!DispatchClientMessage+0xc3
28 00000000`0028ed80 00000000`776211f5 user32!_fnDWORD+0×2d
29 00000000`0028ede0 00000000`774e6e5a ntdll!KiUserCallbackDispatcherContinue
2a 00000000`0028ee68 00000000`774e6e6c user32!NtUserDispatchMessage+0xa
2b 00000000`0028ee70 00000000`774e67c2 user32!DispatchMessageWorker+0×55b
2c 00000000`0028eef0 000007fe`fbcc34a4 user32!IsDialogMessageW+0×153
2d 00000000`0028ef80 000007fe`fbcc583f comctl32!Prop_IsDialogMessage+0×1f0
2e 00000000`0028eff0 000007fe`fbcc5c05 comctl32!_RealPropertySheet+0×31b
2f 00000000`0028f0c0 000007fe`ff214e68 comctl32!_PropertySheet+0×55
30 00000000`0028f100 000007fe`ff214bb1 comdlg32!Print_InvokePropertySheets+0×2c6
31 00000000`0028f660 000007fe`ff21499c comdlg32!PrintDlgExX+0×2be
32 00000000`0028f6c0 00000000`ffec250f comdlg32!PrintDlgExW+0×38
33 00000000`0028f730 00000000`ffec242b notepad!GetPrinterDCviaDialog+0xab
34 00000000`0028f7f0 00000000`ffec23e8 notepad!PrintIt+0×46
35 00000000`0028fb70 00000000`ffec14eb notepad!NPCommand+0xdb
36 00000000`0028fca0 00000000`774e9bd1 notepad!NPWndProc+0×540
37 00000000`0028fce0 00000000`774e98da user32!UserCallWinProcCheckWow+0×1ad
38 00000000`0028fda0 00000000`ffec10bc user32!DispatchMessageWorker+0×3b5
39 00000000`0028fe20 00000000`ffec133c notepad!WinMain+0×16f
3a 00000000`0028fea0 00000000`773c59ed notepad!DisplayNonGenuineDlgWorker+0×2da
3b 00000000`0028ff60 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd
3c 00000000`0028ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see the thread is active, in the middle of the function. For comparison, the next 2 threads are waiting and calling API respectively:

0:000> ~1k
# Child-SP          RetAddr           Call Site
00 00000000`02edf748 00000000`775eb037 ntdll!NtWaitForMultipleObjects+0xa
01 00000000`02edf750 00000000`773c59ed ntdll!TppWaiterpThread+0×14d
02 00000000`02edf9f0 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd
03 00000000`02edfa20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ~2k
# Child-SP          RetAddr           Call Site
00 00000000`0344e048 000007fe`fd3f403e ntdll!NtUnmapViewOfSection+0xa
01 00000000`0344e050 00000000`774e2edf KERNELBASE!FreeLibrary+0xa4
02 00000000`0344e080 000007fe`fdddaab3 user32!PrivateExtractIconsW+0×34b
03 00000000`0344e5a0 000007fe`fdddac28 shell32!SHPrivateExtractIcons+0×50a
04 00000000`0344e870 000007fe`fdde34b4 shell32!SHDefExtractIconW+0×254
05 00000000`0344eb60 000007fe`fdde3435 shell32!CExtractIcon::_ExtractW+0xcd
06 00000000`0344ebe0 000007fe`fdf0d529 shell32!CExtractIconBase::Extract+0×21
07 00000000`0344ec20 000007fe`fdf0d2da shell32!IExtractIcon_Extract+0×43
08 00000000`0344ec60 000007fe`fdddfff0 shell32!_GetILIndexGivenPXIcon+0×22e
09 00000000`0344f100 000007fe`fdde27a4 shell32!_GetILIndexFromItem+0×87
0a 00000000`0344f1a0 000007fe`fddb6506 shell32!SHGetIconIndexFromPIDL+0×66
0b 00000000`0344f1d0 000007fe`fdedb186 shell32!MapIDListToIconILIndex+0×52
0c 00000000`0344f250 000007fe`fdddc54c shell32!CLoadSystemIconTask::InternalResumeRT+0×110
0d 00000000`0344f2e0 000007fe`fde0efcb shell32!CRunnableTask::Run+0xda
0e 00000000`0344f310 000007fe`fde12b56 shell32!CShellTask::TT_Run+0×124
0f 00000000`0344f340 000007fe`fde12cb2 shell32!CShellTaskThread::ThreadProc+0×1d2
10 00000000`0344f3e0 000007fe`ff2b3843 shell32!CShellTaskThread::s_ThreadProc+0×22
11 00000000`0344f410 00000000`775f15db shlwapi!ExecuteWorkItemThreadProc+0xf
12 00000000`0344f440 00000000`775f0c56 ntdll!RtlpTpWorkCallback+0×16b
13 00000000`0344f520 00000000`773c59ed ntdll!TppWorkerThread+0×5ff
14 00000000`0344f820 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd
15 00000000`0344f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub ntdll!NtUnmapViewOfSection+0xa
ntdll!NtAccessCheckAndAuditAlarm:
00000000`77621540 4c8bd1          mov     r10,rcx
00000000`77621543 b826000000      mov     eax,26h
00000000`77621548 0f05            syscall
00000000`7762154a c3              ret
00000000`7762154b 0f1f440000      nop     dword ptr [rax+rax]
ntdll!NtUnmapViewOfSection:
00000000`77621550 4c8bd1          mov     r10,rcx
00000000`77621553 b827000000      mov     eax,27h
00000000`77621558 0f05            syscall

Our Active Thread is not Spiking Thread since CPU consumption is minimal:

0:000> !runaway f
User Mode Time
Thread       Time
0:1ca4      0 days 0:00:00.171
11:1fb0      0 days 0:00:00.000
10:f98       0 days 0:00:00.000
9:eb8       0 days 0:00:00.000
8:1b80      0 days 0:00:00.000
7:139c      0 days 0:00:00.000
6:1d9c      0 days 0:00:00.000
5:1b44      0 days 0:00:00.000
4:1edc      0 days 0:00:00.000
3:830       0 days 0:00:00.000
2:1638      0 days 0:00:00.000
1:1ab0      0 days 0:00:00.000
Kernel Mode Time
Thread       Time
0:1ca4      0 days 0:00:00.421
11:1fb0      0 days 0:00:00.000
10:f98       0 days 0:00:00.000
9:eb8       0 days 0:00:00.000
8:1b80      0 days 0:00:00.000
7:139c      0 days 0:00:00.000
6:1d9c      0 days 0:00:00.000
5:1b44      0 days 0:00:00.000
4:1edc      0 days 0:00:00.000
3:830       0 days 0:00:00.000
2:1638      0 days 0:00:00.000
1:1ab0      0 days 0:00:00.000
Elapsed Time
Thread       Time
11:1fb0      24692 days 13:29:46.335
0:1ca4      0 days 0:02:39.671
1:1ab0      0 days 0:01:48.239
2:1638      0 days 0:01:18.837
4:1edc      0 days 0:01:18.697
3:830       0 days 0:01:18.697
5:1b44      0 days 0:01:18.497
6:1d9c      0 days 0:01:18.387
7:139c      0 days 0:01:15.957
8:1b80      0 days 0:01:14.397
9:eb8       0 days 0:01:01.485
10:f98       0 days 0:00:39.849

However, the huge Elapsed Time for the thread #11 (most likely the value is uninitialized) and its stack trace suggest that the dump was saved on Create Thread debugging event by a debugger:

0:000> ~11k
# Child-SP          RetAddr           Call Site
00 00000000`0666fb08 00000000`00000000 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 231)

Monday, October 26th, 2015

One of the customers of Software Diagnostics Services submitted memory dumps saved by DebugDiag to accompany software logs for the analysis of sudden process exit. We didn’t request such memory dumps and initially dismissed them. However, during software log analysis we decided to look at Adjoint Spaces to see whether there was some additional information in stack traces. We found out that those dumps were saved on each thread exit event. Since other threads were either waiting or Active Threads their analysis gave clues of process behavior before process exit. For example, we found ALPC Wait Chain to Coupled Process. The latter prompted us to analyze Coupled Activities in the software log and diagnose the possible problem there. Since saving memory dumps on thread creation and exit can be useful technique we decided to add Evental Dumps memory analysis pattern to our pattern catalog.

To illustrate this pattern we show Stack Trace Collection from notepad.exe. This process usually has just one thread. But, if we try to open a Print dialog the number of threads increases up to 12.

We attach WinDbg to notepad.exe process and set up debugging event filter (Debug \ Event Filters… menu) for Create Thread event with a command line as shown in this picture:

We then resume execution using g command and switch to Notepad. There we first open File \ Page Setup… dialog. We observe that a memory dump is saved. Then we open File \ Print… dialog and notice the creation 11 more process memory dump:

We now show stack traces from these dumps where we use ~*kc WinDbg command to minimize the amount of inessential for our purposes output:

// 1st dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 ntdll!NtAlertThread
01 ntdll!TppWaiterEnqueueTransition
02 ntdll!TppWaitpSet
03 ntdll!TpSetWait
04 ntdll!TppTimerpInitTimerQueueQueue
05 ntdll!TppTimerpAllocTimerQueue
06 ntdll!TppTimerpAcquirePoolTimerQueue
07 ntdll!TppTimerAlloc
08 ntdll!TpAllocTimer
09 KERNELBASE!CreateThreadpoolTimer
0a rpcrt4!RPC_THREAD_POOL::CreateTimer
0b rpcrt4!GarbageCollectionNeeded
0c rpcrt4!LRPC_CASSOCIATION::RemoveReference
0d rpcrt4!LRPC_CCALL::`vector deleting destructor'
0e rpcrt4!LRPC_CCALL::FreeBuffer
0f rpcrt4!Ndr64pClientFinally
10 rpcrt4!NdrpClientCall3
11 rpcrt4!NdrClientCall3
12 sechost!LsaLookupClose
13 sechost!LookupAccountNameInternal
14 sechost!LookupAccountNameLocalW
15 rpcrt4!RpcpLookupAccountNameDirect
16 rpcrt4!RpcpLookupAccountName
17 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation
18 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation
19 rpcrt4!RpcBindingSetAuthInfoExW
1a winspool!STRING_HANDLE_bind
1b rpcrt4!GenericHandleMgr
1c rpcrt4!ExplicitBindHandleMgr
1d rpcrt4!Ndr64pClientSetupTransferSyntax
1e rpcrt4!NdrpClientCall3
1f rpcrt4!NdrClientCall3
20 winspool!RpcSplOpenPrinter
21 winspool!OpenPrinterRPC
22 winspool!OpenPrinter2W
23 comdlg32!PrintOpenPrinter
24 comdlg32!PrintDlgX
25 comdlg32!PageSetupDlgX
26 comdlg32!PageSetupDlgW
27 notepad!NPCommand
28 notepad!NPWndProc
29 user32!UserCallWinProcCheckWow
2a user32!DispatchMessageWorker
2b notepad!WinMain
2c notepad!DisplayNonGenuineDlgWorker
2d kernel32!BaseThreadInitThunk
2e ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 2nd dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 ntdll!ZwOpenKeyEx
01 kernel32!LocalBaseRegOpenKey
02 kernel32!RegOpenKeyExInternalW
03 kernel32!RegOpenKeyExW
04 rpcrt4!Server2003NegotiateDisable
05 rpcrt4!IsBindTimeFeatureNegotiationDisabled
06 rpcrt4!OSF_CCONNECTION::SendBindPacket
07 rpcrt4!OSF_CCONNECTION::ActuallyDoBinding
08 rpcrt4!OSF_CCONNECTION::OpenConnectionAndBind
09 rpcrt4!OSF_CCALL::BindToServer
0a rpcrt4!OSF_BINDING_HANDLE::InitCCallWithAssociation
0b rpcrt4!OSF_BINDING_HANDLE::AllocateCCall
0c rpcrt4!OSF_BINDING_HANDLE::NegotiateTransferSyntax
0d rpcrt4!I_RpcNegotiateTransferSyntax
0e rpcrt4!Ndr64pClientSetupTransferSyntax
0f rpcrt4!NdrpClientCall3
10 rpcrt4!NdrClientCall3
11 srvcli!NetShareEnum
12 ntshrui!CShareCache::RefreshNoCritSec
13 ntshrui!CShareCache::Refresh
14 ntshrui!DllMain
15 ntdll!RtlRunOnceExecuteOnce
16 kernel32!InitOnceExecuteOnce
17 ntshrui!DllGetClassObject
18 ole32!CClassCache::CDllPathEntry::DllGetClassObject
19 ole32!CClassCache::CDllFnPtrMoniker::BindToObjectNoSwitch
1a ole32!CClassCache::GetClassObject
1b ole32!CServerContextActivator::CreateInstance
1c ole32!ActivationPropertiesIn::DelegateCreateInstance
1d ole32!CApartmentActivator::CreateInstance
1e ole32!CProcessActivator::CCICallback
1f ole32!CProcessActivator::AttemptActivation
20 ole32!CProcessActivator::ActivateByContext
21 ole32!CProcessActivator::CreateInstance
22 ole32!ActivationPropertiesIn::DelegateCreateInstance
23 ole32!CClientContextActivator::CreateInstance
24 ole32!ActivationPropertiesIn::DelegateCreateInstance
25 ole32!ICoCreateInstanceEx
26 ole32!CoCreateInstance
27 shell32!_SHCoCreateInstance
28 shell32!SHExtCoCreateInstance
29 shell32!DCA_SHExtCoCreateInstance
2a shell32!CFSIconOverlayManager::_s_LoadIconOverlayIdentifiers
2b shell32!CFSIconOverlayManager::CreateInstance
2c shell32!IconOverlayManagerInit
2d shell32!GetIconOverlayManager
2e shell32!FileIconInit
2f shell32!Shell_GetImageLists
30 comdlg32!CPrintDialog::CPrintDialog
31 comdlg32!Print_GeneralDlgProc
32 user32!UserCallDlgProcCheckWow
33 user32!DefDlgProcWorker
34 user32!InternalCreateDialog
35 user32!CreateDialogIndirectParamAorW
36 user32!CreateDialogIndirectParamW
37 comctl32!_CreatePageDialog
38 comctl32!_CreatePage
39 comctl32!PageChange
3a comctl32!InitPropSheetDlg
3b comctl32!PropSheetDlgProc
3c user32!UserCallDlgProcCheckWow
3d user32!DefDlgProcWorker
3e user32!InternalCreateDialog
3f user32!CreateDialogIndirectParamAorW
40 user32!CreateDialogIndirectParamW
41 comctl32!_RealPropertySheet
42 comctl32!_PropertySheet
43 comdlg32!Print_InvokePropertySheets
44 comdlg32!PrintDlgExX
45 comdlg32!PrintDlgExW
46 notepad!GetPrinterDCviaDialog
47 notepad!PrintIt
48 notepad!NPCommand
49 notepad!NPWndProc
4a user32!UserCallWinProcCheckWow
4b user32!DispatchMessageWorker
4c notepad!WinMain
4d notepad!DisplayNonGenuineDlgWorker
4e kernel32!BaseThreadInitThunk
4f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 3rd dump: we see 2 threads start at the same time

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 ntdll!RtlCompareMemoryUlong
01 ntdll!RtlpAllocateHeap
02 ntdll!RtlAllocateHeap
03 ntdll!RtlDebugAllocateHeap
04 ntdll! ?? ::FNODOBFM::`string'
05 ntdll!RtlAllocateHeap
06 ole32!CRpcResolver::GetThreadWinstaDesktop
07 ole32!CRpcResolver::GetConnection
08 ole32!CoInitializeSecurity
09 ole32!InitializeSecurity
0a ole32!ChannelProcessInitialize
0b ole32!CComApartment::InitRemoting
0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp
0d ole32!CGIPTable::RegisterInterfaceInGlobal
0e shell32!MarshalToGIT
0f shell32!CBrowserProgressAggregator::BeginSession
10 shell32!IUnknown_BeginBrowserProgressSession
11 shell32!CDefView::CreateViewWindow3
12 shell32!CExplorerBrowser::_CreateViewWindow
13 shell32!CExplorerBrowser::_SwitchView
14 shell32!CExplorerBrowser::_BrowseToView
15 shell32!CExplorerBrowser::_BrowseObjectInternal
16 shell32!CExplorerBrowser::_OnBrowseObject
17 shell32!CExplorerBrowser::BrowseObject
18 comdlg32!CPrintDialog::CreatePrintBrowser
19 comdlg32!CPrintDialog::OnInitDialog
1a comdlg32!Print_GeneralDlgProc
1b user32!UserCallDlgProcCheckWow
1c user32!DefDlgProcWorker
1d user32!InternalCreateDialog
1e user32!CreateDialogIndirectParamAorW
1f user32!CreateDialogIndirectParamW
20 comctl32!_CreatePageDialog
21 comctl32!_CreatePage
22 comctl32!PageChange
23 comctl32!InitPropSheetDlg
24 comctl32!PropSheetDlgProc
25 user32!UserCallDlgProcCheckWow
26 user32!DefDlgProcWorker
27 user32!InternalCreateDialog
28 user32!CreateDialogIndirectParamAorW
29 user32!CreateDialogIndirectParamW
2a comctl32!_RealPropertySheet
2b comctl32!_PropertySheet
2c comdlg32!Print_InvokePropertySheets
2d comdlg32!PrintDlgExX
2e comdlg32!PrintDlgExW
2f notepad!GetPrinterDCviaDialog
30 notepad!PrintIt
31 notepad!NPCommand
32 notepad!NPWndProc
33 user32!UserCallWinProcCheckWow
34 user32!DispatchMessageWorker
35 notepad!WinMain
36 notepad!DisplayNonGenuineDlgWorker
37 kernel32!BaseThreadInitThunk
38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 sechost!LsaLookupOpenLocalPolicy
05 sechost!LookupAccountNameInternal
06 sechost!LookupAccountNameLocalW
07 rpcrt4!RpcpLookupAccountNameDirect
08 rpcrt4!RpcpLookupAccountName
09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation
0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation
0b rpcrt4!RpcBindingSetAuthInfoExW
0c winspool!STRING_HANDLE_bind
0d rpcrt4!GenericHandleMgr
0e rpcrt4!ExplicitBindHandleMgr
0f rpcrt4!Ndr64pClientSetupTransferSyntax
10 rpcrt4!NdrpClientCall3
11 rpcrt4!NdrClientCall3
12 winspool!RpcSplOpenPrinter
13 winspool!OpenPrinterRPC
14 winspool!OpenPrinter2W
15 prncache!PrintCache::Listeners::Listener::Start
16 prncache!PrintCache::Listeners::Listener::StartCB
17 ntdll!TppWorkpExecuteCallback
18 ntdll!TppWorkerThread
19 kernel32!BaseThreadInitThunk
1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 0 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 4th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 ntdll!RtlCompareMemoryUlong
01 ntdll!RtlpAllocateHeap
02 ntdll!RtlAllocateHeap
03 ntdll!RtlDebugAllocateHeap
04 ntdll! ?? ::FNODOBFM::`string'
05 ntdll!RtlAllocateHeap
06 ole32!CRpcResolver::GetThreadWinstaDesktop
07 ole32!CRpcResolver::GetConnection
08 ole32!CoInitializeSecurity
09 ole32!InitializeSecurity
0a ole32!ChannelProcessInitialize
0b ole32!CComApartment::InitRemoting
0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp
0d ole32!CGIPTable::RegisterInterfaceInGlobal
0e shell32!MarshalToGIT
0f shell32!CBrowserProgressAggregator::BeginSession
10 shell32!IUnknown_BeginBrowserProgressSession
11 shell32!CDefView::CreateViewWindow3
12 shell32!CExplorerBrowser::_CreateViewWindow
13 shell32!CExplorerBrowser::_SwitchView
14 shell32!CExplorerBrowser::_BrowseToView
15 shell32!CExplorerBrowser::_BrowseObjectInternal
16 shell32!CExplorerBrowser::_OnBrowseObject
17 shell32!CExplorerBrowser::BrowseObject
18 comdlg32!CPrintDialog::CreatePrintBrowser
19 comdlg32!CPrintDialog::OnInitDialog
1a comdlg32!Print_GeneralDlgProc
1b user32!UserCallDlgProcCheckWow
1c user32!DefDlgProcWorker
1d user32!InternalCreateDialog
1e user32!CreateDialogIndirectParamAorW
1f user32!CreateDialogIndirectParamW
20 comctl32!_CreatePageDialog
21 comctl32!_CreatePage
22 comctl32!PageChange
23 comctl32!InitPropSheetDlg
24 comctl32!PropSheetDlgProc
25 user32!UserCallDlgProcCheckWow
26 user32!DefDlgProcWorker
27 user32!InternalCreateDialog
28 user32!CreateDialogIndirectParamAorW
29 user32!CreateDialogIndirectParamW
2a comctl32!_RealPropertySheet
2b comctl32!_PropertySheet
2c comdlg32!Print_InvokePropertySheets
2d comdlg32!PrintDlgExX
2e comdlg32!PrintDlgExW
2f notepad!GetPrinterDCviaDialog
30 notepad!PrintIt
31 notepad!NPCommand
32 notepad!NPWndProc
33 user32!UserCallWinProcCheckWow
34 user32!DispatchMessageWorker
35 notepad!WinMain
36 notepad!DisplayNonGenuineDlgWorker
37 kernel32!BaseThreadInitThunk
38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 sechost!LsaLookupOpenLocalPolicy
05 sechost!LookupAccountNameInternal
06 sechost!LookupAccountNameLocalW
07 rpcrt4!RpcpLookupAccountNameDirect
08 rpcrt4!RpcpLookupAccountName
09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation
0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation
0b rpcrt4!RpcBindingSetAuthInfoExW
0c winspool!STRING_HANDLE_bind
0d rpcrt4!GenericHandleMgr
0e rpcrt4!ExplicitBindHandleMgr
0f rpcrt4!Ndr64pClientSetupTransferSyntax
10 rpcrt4!NdrpClientCall3
11 rpcrt4!NdrClientCall3
12 winspool!RpcSplOpenPrinter
13 winspool!OpenPrinterRPC
14 winspool!OpenPrinter2W
15 prncache!PrintCache::Listeners::Listener::Start
16 prncache!PrintCache::Listeners::Listener::StartCB
17 ntdll!TppWorkpExecuteCallback
18 ntdll!TppWorkerThread
19 kernel32!BaseThreadInitThunk
1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!LdrpInitializeThread
01 ntdll!LdrpInitialize
02 ntdll!LdrInitializeThunk

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 5th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 sechost!LsaLookupClose
05 sechost!LookupAccountNameInternal
06 sechost!LookupAccountNameLocalW
07 rpcrt4!RpcpLookupAccountNameDirect
08 rpcrt4!RpcpLookupAccountName
09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation
0a rpcrt4!LRPC_FAST_BINDING_HANDLE::SetAuthInformation
0b rpcrt4!LRPC_FAST_BINDING_HANDLE::LRPC_FAST_BINDING_HANDLE
0c rpcrt4!LrpcCreateFastBindingHandle
0d rpcrt4!RpcBindingCreateW
0e ole32!CFastBH::CreateFromBindingString
0f ole32!CFastBH::GetOrCreate
10 ole32!CRpcResolver::GetConnection
11 ole32!CoInitializeSecurity
12 ole32!InitializeSecurity
13 ole32!ChannelProcessInitialize
14 ole32!CComApartment::InitRemoting
15 ole32!CGIPTable::RegisterInterfaceInGlobalHlp
16 ole32!CGIPTable::RegisterInterfaceInGlobal
17 shell32!MarshalToGIT
18 shell32!CBrowserProgressAggregator::BeginSession
19 shell32!IUnknown_BeginBrowserProgressSession
1a shell32!CDefView::CreateViewWindow3
1b shell32!CExplorerBrowser::_CreateViewWindow
1c shell32!CExplorerBrowser::_SwitchView
1d shell32!CExplorerBrowser::_BrowseToView
1e shell32!CExplorerBrowser::_BrowseObjectInternal
1f shell32!CExplorerBrowser::_OnBrowseObject
20 shell32!CExplorerBrowser::BrowseObject
21 comdlg32!CPrintDialog::CreatePrintBrowser
22 comdlg32!CPrintDialog::OnInitDialog
23 comdlg32!Print_GeneralDlgProc
24 user32!UserCallDlgProcCheckWow
25 user32!DefDlgProcWorker
26 user32!InternalCreateDialog
27 user32!CreateDialogIndirectParamAorW
28 user32!CreateDialogIndirectParamW
29 comctl32!_CreatePageDialog
2a comctl32!_CreatePage
2b comctl32!PageChange
2c comctl32!InitPropSheetDlg
2d comctl32!PropSheetDlgProc
2e user32!UserCallDlgProcCheckWow
2f user32!DefDlgProcWorker
30 user32!InternalCreateDialog
31 user32!CreateDialogIndirectParamAorW
32 user32!CreateDialogIndirectParamW
33 comctl32!_RealPropertySheet
34 comctl32!_PropertySheet
35 comdlg32!Print_InvokePropertySheets
36 comdlg32!PrintDlgExX
37 comdlg32!PrintDlgExW
38 notepad!GetPrinterDCviaDialog
39 notepad!PrintIt
3a notepad!NPCommand
3b notepad!NPWndProc
3c user32!UserCallWinProcCheckWow
3d user32!DispatchMessageWorker
3e notepad!WinMain
3f notepad!DisplayNonGenuineDlgWorker
40 kernel32!BaseThreadInitThunk
41 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 winspool!RpcSplOpenPrinter
05 winspool!OpenPrinterRPC
06 winspool!OpenPrinter2W
07 prncache!PrintCache::Listeners::Listener::Start
08 prncache!PrintCache::Listeners::Listener::StartCB
09 ntdll!TppWorkpExecuteCallback
0a ntdll!TppWorkerThread
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 winspool!RpcEnumPrinters
05 winspool!EnumPrintersW
06 prncache!PrintCache::Listeners::ConnectionListener::EnumConnectionsAndRegister
07 prncache!PrintCache::Listeners::ConnectionListener::UpdateCB
08 ntdll!TppWorkpExecuteCallback
09 ntdll!TppWorkerThread
0a kernel32!BaseThreadInitThunk
0b ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 6th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 ntdll!NtAlpcConnectPort
01 rpcrt4!LRPC_CASSOCIATION::AlpcConnect
02 rpcrt4!LRPC_CASSOCIATION::Connect
03 rpcrt4!LRPC_BASE_BINDING_HANDLE::DriveStateForward
04 rpcrt4!LRPC_FAST_BINDING_HANDLE::Bind
05 rpcrt4!RpcBindingBind
06 ole32!CFastBH::CreateFromBindingString
07 ole32!CFastBH::GetOrCreate
08 ole32!CRpcResolver::GetConnection
09 ole32!CoInitializeSecurity
0a ole32!InitializeSecurity
0b ole32!ChannelProcessInitialize
0c ole32!CComApartment::InitRemoting
0d ole32!CGIPTable::RegisterInterfaceInGlobalHlp
0e ole32!CGIPTable::RegisterInterfaceInGlobal
0f shell32!MarshalToGIT
10 shell32!CBrowserProgressAggregator::BeginSession
11 shell32!IUnknown_BeginBrowserProgressSession
12 shell32!CDefView::CreateViewWindow3
13 shell32!CExplorerBrowser::_CreateViewWindow
14 shell32!CExplorerBrowser::_SwitchView
15 shell32!CExplorerBrowser::_BrowseToView
16 shell32!CExplorerBrowser::_BrowseObjectInternal
17 shell32!CExplorerBrowser::_OnBrowseObject
18 shell32!CExplorerBrowser::BrowseObject
19 comdlg32!CPrintDialog::CreatePrintBrowser
1a comdlg32!CPrintDialog::OnInitDialog
1b comdlg32!Print_GeneralDlgProc
1c user32!UserCallDlgProcCheckWow
1d user32!DefDlgProcWorker
1e user32!InternalCreateDialog
1f user32!CreateDialogIndirectParamAorW
20 user32!CreateDialogIndirectParamW
21 comctl32!_CreatePageDialog
22 comctl32!_CreatePage
23 comctl32!PageChange
24 comctl32!InitPropSheetDlg
25 comctl32!PropSheetDlgProc
26 user32!UserCallDlgProcCheckWow
27 user32!DefDlgProcWorker
28 user32!InternalCreateDialog
29 user32!CreateDialogIndirectParamAorW
2a user32!CreateDialogIndirectParamW
2b comctl32!_RealPropertySheet
2c comctl32!_PropertySheet
2d comdlg32!Print_InvokePropertySheets
2e comdlg32!PrintDlgExX
2f comdlg32!PrintDlgExW
30 notepad!GetPrinterDCviaDialog
31 notepad!PrintIt
32 notepad!NPCommand
33 notepad!NPWndProc
34 user32!UserCallWinProcCheckWow
35 user32!DispatchMessageWorker
36 notepad!WinMain
37 notepad!DisplayNonGenuineDlgWorker
38 kernel32!BaseThreadInitThunk
39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!RtlEnterCriticalSection
01 ntdll!RtlDebugAllocateHeap
02 ntdll! ?? ::FNODOBFM::`string'
03 ntdll!RtlAllocateHeap
04 rpcrt4!AllocWrapper
05 rpcrt4!SID_CACHE::Query
06 rpcrt4!RpcpLookupAccountName
07 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation
08 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation
09 rpcrt4!RpcBindingSetAuthInfoExW
0a winspool!STRING_HANDLE_bind
0b rpcrt4!GenericHandleMgr
0c rpcrt4!ExplicitBindHandleMgr
0d rpcrt4!Ndr64pClientSetupTransferSyntax
0e rpcrt4!NdrpClientCall3
0f rpcrt4!NdrClientCall3
10 winspool!RpcSplOpenPrinter
11 winspool!OpenPrinterRPC
12 winspool!OpenPrinter2W
13 prncache!PrintCache::Listeners::Listener::Start
14 prncache!PrintCache::Listeners::Listener::StartCB
15 ntdll!TppWorkpExecuteCallback
16 ntdll!TppWorkerThread
17 kernel32!BaseThreadInitThunk
18 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 user32!NtUserAttachThreadInput
01 shell32!CWaitTask::s_WaitBeforeCursing
02 ntdll!RtlpTpWaitCallback
03 ntdll!TppWaitpExecuteCallback
04 ntdll!TppWorkerThread
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 7th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 rpcrt4!RpcStringFreeW
01 ole32!CFastBH::CreateFromBindingString
02 ole32!CFastBH::GetOrCreate
03 ole32!CRpcResolver::GetConnection
04 ole32!CoInitializeSecurity
05 ole32!InitializeSecurity
06 ole32!ChannelProcessInitialize
07 ole32!CComApartment::InitRemoting
08 ole32!CGIPTable::RegisterInterfaceInGlobalHlp
09 ole32!CGIPTable::RegisterInterfaceInGlobal
0a shell32!MarshalToGIT
0b shell32!CBrowserProgressAggregator::BeginSession
0c shell32!IUnknown_BeginBrowserProgressSession
0d shell32!CDefView::CreateViewWindow3
0e shell32!CExplorerBrowser::_CreateViewWindow
0f shell32!CExplorerBrowser::_SwitchView
10 shell32!CExplorerBrowser::_BrowseToView
11 shell32!CExplorerBrowser::_BrowseObjectInternal
12 shell32!CExplorerBrowser::_OnBrowseObject
13 shell32!CExplorerBrowser::BrowseObject
14 comdlg32!CPrintDialog::CreatePrintBrowser
15 comdlg32!CPrintDialog::OnInitDialog
16 comdlg32!Print_GeneralDlgProc
17 user32!UserCallDlgProcCheckWow
18 user32!DefDlgProcWorker
19 user32!InternalCreateDialog
1a user32!CreateDialogIndirectParamAorW
1b user32!CreateDialogIndirectParamW
1c comctl32!_CreatePageDialog
1d comctl32!_CreatePage
1e comctl32!PageChange
1f comctl32!InitPropSheetDlg
20 comctl32!PropSheetDlgProc
21 user32!UserCallDlgProcCheckWow
22 user32!DefDlgProcWorker
23 user32!InternalCreateDialog
24 user32!CreateDialogIndirectParamAorW
25 user32!CreateDialogIndirectParamW
26 comctl32!_RealPropertySheet
27 comctl32!_PropertySheet
28 comdlg32!Print_InvokePropertySheets
29 comdlg32!PrintDlgExX
2a comdlg32!PrintDlgExW
2b notepad!GetPrinterDCviaDialog
2c notepad!PrintIt
2d notepad!NPCommand
2e notepad!NPWndProc
2f user32!UserCallWinProcCheckWow
30 user32!DispatchMessageWorker
31 notepad!WinMain
32 notepad!DisplayNonGenuineDlgWorker
33 kernel32!BaseThreadInitThunk
34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 winspool!RpcSplOpenPrinter
05 winspool!OpenPrinterRPC
06 winspool!OpenPrinter2W
07 prncache!PrintCache::Listeners::Listener::Start
08 prncache!PrintCache::Listeners::Listener::StartCB
09 ntdll!TppWorkpExecuteCallback
0a ntdll!TppWorkerThread
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjectsExImplementation
03 user32!RealMsgWaitForMultipleObjectsEx
04 user32!MsgWaitForMultipleObjectsEx
05 user32!MsgWaitForMultipleObjects
06 shell32!SHProcessMessagesUntilEventsEx
07 shell32!CWaitTask::s_WaitBeforeCursing
08 ntdll!RtlpTpWaitCallback
09 ntdll!TppWaitpExecuteCallback
0a ntdll!TppWorkerThread
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 8th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 user32!NtUserSetWindowLongPtr
01 user32!SetWindowLongPtr
02 ole32!OXIDEntry::StartServer
03 ole32!CGIPTable::RegisterInterfaceInGlobalHlp
04 ole32!CGIPTable::RegisterInterfaceInGlobal
05 shell32!MarshalToGIT
06 shell32!CBrowserProgressAggregator::BeginSession
07 shell32!IUnknown_BeginBrowserProgressSession
08 shell32!CDefView::CreateViewWindow3
09 shell32!CExplorerBrowser::_CreateViewWindow
0a shell32!CExplorerBrowser::_SwitchView
0b shell32!CExplorerBrowser::_BrowseToView
0c shell32!CExplorerBrowser::_BrowseObjectInternal
0d shell32!CExplorerBrowser::_OnBrowseObject
0e shell32!CExplorerBrowser::BrowseObject
0f comdlg32!CPrintDialog::CreatePrintBrowser
10 comdlg32!CPrintDialog::OnInitDialog
11 comdlg32!Print_GeneralDlgProc
12 user32!UserCallDlgProcCheckWow
13 user32!DefDlgProcWorker
14 user32!InternalCreateDialog
15 user32!CreateDialogIndirectParamAorW
16 user32!CreateDialogIndirectParamW
17 comctl32!_CreatePageDialog
18 comctl32!_CreatePage
19 comctl32!PageChange
1a comctl32!InitPropSheetDlg
1b comctl32!PropSheetDlgProc
1c user32!UserCallDlgProcCheckWow
1d user32!DefDlgProcWorker
1e user32!InternalCreateDialog
1f user32!CreateDialogIndirectParamAorW
20 user32!CreateDialogIndirectParamW
21 comctl32!_RealPropertySheet
22 comctl32!_PropertySheet
23 comdlg32!Print_InvokePropertySheets
24 comdlg32!PrintDlgExX
25 comdlg32!PrintDlgExW
26 notepad!GetPrinterDCviaDialog
27 notepad!PrintIt
28 notepad!NPCommand
29 notepad!NPWndProc
2a user32!UserCallWinProcCheckWow
2b user32!DispatchMessageWorker
2c notepad!WinMain
2d notepad!DisplayNonGenuineDlgWorker
2e kernel32!BaseThreadInitThunk
2f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjectsExImplementation
03 user32!RealMsgWaitForMultipleObjectsEx
04 user32!MsgWaitForMultipleObjectsEx
05 user32!MsgWaitForMultipleObjects
06 shell32!SHProcessMessagesUntilEventsEx
07 shell32!CWaitTask::s_WaitBeforeCursing
08 ntdll!RtlpTpWaitCallback
09 ntdll!TppWaitpExecuteCallback
0a ntdll!TppWorkerThread
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 9th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 user32!NtUserPeekMessage
01 user32!PeekMessageW
02 shell32!PeekMessageWithWakeMask
03 shell32!SHProcessMessagesUntilEventsEx
04 shell32!CDefView::_SetItemCollection
05 shell32!CDefView::_CreateNewCollection
06 shell32!CDefView::CreateViewWindow3
07 shell32!CExplorerBrowser::_CreateViewWindow
08 shell32!CExplorerBrowser::_SwitchView
09 shell32!CExplorerBrowser::_BrowseToView
0a shell32!CExplorerBrowser::_BrowseObjectInternal
0b shell32!CExplorerBrowser::_OnBrowseObject
0c shell32!CExplorerBrowser::BrowseObject
0d comdlg32!CPrintDialog::CreatePrintBrowser
0e comdlg32!CPrintDialog::OnInitDialog
0f comdlg32!Print_GeneralDlgProc
10 user32!UserCallDlgProcCheckWow
11 user32!DefDlgProcWorker
12 user32!InternalCreateDialog
13 user32!CreateDialogIndirectParamAorW
14 user32!CreateDialogIndirectParamW
15 comctl32!_CreatePageDialog
16 comctl32!_CreatePage
17 comctl32!PageChange
18 comctl32!InitPropSheetDlg
19 comctl32!PropSheetDlgProc
1a user32!UserCallDlgProcCheckWow
1b user32!DefDlgProcWorker
1c user32!InternalCreateDialog
1d user32!CreateDialogIndirectParamAorW
1e user32!CreateDialogIndirectParamW
1f comctl32!_RealPropertySheet
20 comctl32!_PropertySheet
21 comdlg32!Print_InvokePropertySheets
22 comdlg32!PrintDlgExX
23 comdlg32!PrintDlgExW
24 notepad!GetPrinterDCviaDialog
25 notepad!PrintIt
26 notepad!NPCommand
27 notepad!NPWndProc
28 user32!UserCallWinProcCheckWow
29 user32!DispatchMessageWorker
2a notepad!WinMain
2b notepad!DisplayNonGenuineDlgWorker
2c kernel32!BaseThreadInitThunk
2d ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!NtWaitForSingleObject
01 KERNELBASE!WaitForSingleObjectEx
02 shlwapi!CreateThreadWorker
03 shlwapi!SHCreateThread
04 shell32!CEnumThread::Run
05 shell32!CEnumTask::_StartEnumThread
06 shell32!CEnumTask::_IncrEnumFolder
07 shell32!CEnumTask::InternalResumeRT
08 shell32!CRunnableTask::Run
09 shell32!CShellTask::TT_Run
0a shell32!CShellTaskThread::ThreadProc
0b shell32!CShellTaskThread::s_ThreadProc
0c shlwapi!ExecuteWorkItemThreadProc
0d ntdll!RtlpTpWorkCallback
0e ntdll!TppWorkerThread
0f kernel32!BaseThreadInitThunk
10 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjectsExImplementation
03 user32!RealMsgWaitForMultipleObjectsEx
04 user32!MsgWaitForMultipleObjectsEx
05 user32!MsgWaitForMultipleObjects
06 shell32!SHProcessMessagesUntilEventsEx
07 shell32!CWaitTask::s_WaitBeforeCursing
08 ntdll!RtlpTpWaitCallback
09 ntdll!TppWaitpExecuteCallback
0a ntdll!TppWorkerThread
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
# Call Site
00 ntdll!ZwDelayExecution
01 KERNELBASE!SleepEx
02 ole32!CROIDTable::WorkerThreadLoop
03 ole32!CRpcThread::WorkerLoop
04 ole32!CRpcThreadCache::RpcWorkerThreadEntry
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

9 Id: 1344.1310 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 10th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 kernel32!TlsGetValue
01 usp10!UspFreeMem
02 usp10!ScriptApplyDigitSubstitution
03 lpk!LpkCharsetDraw
04 lpk!LpkDrawTextEx
05 user32!DT_GetExtentMinusPrefixes
06 user32!NeedsEndEllipsis
07 user32!AddEllipsisAndDrawLine
08 user32!DrawTextExWorker
09 user32!DrawTextW
0a comctl32!CLVView::_ComputeLabelSizeWorker
0b comctl32!CLVView::v_RecomputeLabelSize
0c comctl32!CLVListView::v_DrawItem
0d comctl32!CLVDrawItemManager::DrawItem
0e comctl32!CLVDrawManager::_PaintItems
0f comctl32!CLVDrawManager::_PaintWorkArea
10 comctl32!CLVDrawManager::_OnPaintWorkAreas
11 comctl32!CLVDrawManager::_OnPaint
12 comctl32!CListView::WndProc
13 comctl32!CListView::s_WndProc
14 user32!UserCallWinProcCheckWow
15 user32!CallWindowProcAorW
16 user32!CallWindowProcW
17 comctl32!CallOriginalWndProc
18 comctl32!CallNextSubclassProc
19 comctl32!DefSubclassProc
1a shell32!DefSubclassProc
1b shell32!CListViewHost::s_ListViewSubclassWndProc
1c comctl32!CallNextSubclassProc
1d comctl32!MasterSubclassProc
1e user32!UserCallWinProcCheckWow
1f user32!DispatchClientMessage
20 user32!_fnDWORD
21 ntdll!KiUserCallbackDispatcherContinue
22 user32!NtUserDispatchMessage
23 user32!DispatchMessageWorker
24 user32!IsDialogMessageW
25 comctl32!Prop_IsDialogMessage
26 comctl32!_RealPropertySheet
27 comctl32!_PropertySheet
28 comdlg32!Print_InvokePropertySheets
29 comdlg32!PrintDlgExX
2a comdlg32!PrintDlgExW
2b notepad!GetPrinterDCviaDialog
2c notepad!PrintIt
2d notepad!NPCommand
2e notepad!NPWndProc
2f user32!UserCallWinProcCheckWow
30 user32!DispatchMessageWorker
31 notepad!WinMain
32 notepad!DisplayNonGenuineDlgWorker
33 kernel32!BaseThreadInitThunk
34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!ZwCreateSection
01 KERNELBASE!BasepLoadLibraryAsDataFileInternal
02 KERNELBASE!LoadLibraryExW
03 user32!PrivateExtractIconsW
04 shell32!SHPrivateExtractIcons
05 shell32!SHDefExtractIconW
06 shell32!CExtractIcon::_ExtractW
07 shell32!CExtractIconBase::Extract
08 shell32!IExtractIcon_Extract
09 shell32!_GetILIndexGivenPXIcon
0a shell32!_GetILIndexFromItem
0b shell32!SHGetIconIndexFromPIDL
0c shell32!MapIDListToIconILIndex
0d shell32!CLoadSystemIconTask::InternalResumeRT
0e shell32!CRunnableTask::Run
0f shell32!CShellTask::TT_Run
10 shell32!CShellTaskThread::ThreadProc
11 shell32!CShellTaskThread::s_ThreadProc
12 shlwapi!ExecuteWorkItemThreadProc
13 ntdll!RtlpTpWorkCallback
14 ntdll!TppWorkerThread
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!ZwCreateSection
01 KERNELBASE!BasepLoadLibraryAsDataFileInternal
02 KERNELBASE!LoadLibraryExW
03 user32!PrivateExtractIconsW
04 shell32!SHPrivateExtractIcons
05 shell32!SHDefExtractIconW
06 shell32!CExtractIcon::_ExtractW
07 shell32!CExtractIconBase::Extract
08 shell32!IExtractIcon_Extract
09 shell32!_GetILIndexGivenPXIcon
0a shell32!_GetILIndexFromItem
0b shell32!SHGetIconIndexFromPIDL
0c shell32!MapIDListToIconILIndex
0d shell32!CLoadSystemIconTask::InternalResumeRT
0e shell32!CRunnableTask::Run
0f shell32!CShellTask::TT_Run
10 shell32!CShellTaskThread::ThreadProc
11 shell32!CShellTaskThread::s_ThreadProc
12 shlwapi!ExecuteWorkItemThreadProc
13 ntdll!RtlpTpWorkCallback
14 ntdll!TppWorkerThread
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!ZwCreateSection
01 KERNELBASE!BasepLoadLibraryAsDataFileInternal
02 KERNELBASE!LoadLibraryExW
03 user32!PrivateExtractIconsW
04 shell32!SHPrivateExtractIcons
05 shell32!SHDefExtractIconW
06 shell32!CExtractIcon::_ExtractW
07 shell32!CExtractIconBase::Extract
08 shell32!IExtractIcon_Extract
09 shell32!_GetILIndexGivenPXIcon
0a shell32!_GetILIndexFromItem
0b shell32!SHGetIconIndexFromPIDL
0c shell32!MapIDListToIconILIndex
0d shell32!CLoadSystemIconTask::InternalResumeRT
0e shell32!CRunnableTask::Run
0f shell32!CShellTask::TT_Run
10 shell32!CShellTaskThread::ThreadProc
11 shell32!CShellTaskThread::s_ThreadProc
12 shlwapi!ExecuteWorkItemThreadProc
13 ntdll!RtlpTpWorkCallback
14 ntdll!TppWorkerThread
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 ntdll!ZwCreateSection
01 KERNELBASE!BasepLoadLibraryAsDataFileInternal
02 KERNELBASE!LoadLibraryExW
03 user32!PrivateExtractIconsW
04 shell32!SHPrivateExtractIcons
05 shell32!SHDefExtractIconW
06 shell32!CExtractIcon::_ExtractW
07 shell32!CExtractIconBase::Extract
08 shell32!IExtractIcon_Extract
09 shell32!_GetILIndexGivenPXIcon
0a shell32!_GetILIndexFromItem
0b shell32!SHGetIconIndexFromPIDL
0c shell32!MapIDListToIconILIndex
0d shell32!CLoadSystemIconTask::InternalResumeRT
0e shell32!CRunnableTask::Run
0f shell32!CShellTask::TT_Run
10 shell32!CShellTaskThread::ThreadProc
11 shell32!CShellTaskThread::s_ThreadProc
12 shlwapi!ExecuteWorkItemThreadProc
13 ntdll!RtlpTpWorkCallback
14 ntdll!TppWorkerThread
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
# Call Site
00 ntdll!NtQueryKey
01 kernel32!BaseRegGetKeySemantics
02 kernel32!BaseRegGetUserAndMachineClass
03 kernel32!LocalBaseRegQueryValue
04 kernel32!RegQueryValueExW
05 shlwapi!SHRegQueryValueW
06 shlwapi!SHRegGetValueW
07 shlwapi!SHQueryValueExW
08 shell32!_GetServerInfo
09 shell32!_SHCoCreateInstance
0a shell32!CRegFolder::_CreateCachedRegFolder
0b shell32!CRegFolder::_BindToItem
0c shell32!CRegFolder::BindToObject
0d shell32!CRegFolder::_BindToItem
0e shell32!CRegFolder::BindToObject
0f shell32!SHBindToObject
10 shell32!CIconOverlayTask::InternalResumeRT
11 shell32!CRunnableTask::Run
12 shell32!CShellTask::TT_Run
13 shell32!CShellTaskThread::ThreadProc
14 shell32!CShellTaskThread::s_ThreadProc
15 shlwapi!ExecuteWorkItemThreadProc
16 ntdll!RtlpTpWorkCallback
17 ntdll!TppWorkerThread
18 kernel32!BaseThreadInitThunk
19 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
# Call Site
00 ntdll!ZwDelayExecution
01 KERNELBASE!SleepEx
02 ole32!CROIDTable::WorkerThreadLoop
03 ole32!CRpcThread::WorkerLoop
04 ole32!CRpcThreadCache::RpcWorkerThreadEntry
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 11th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 usp10!GenericEngineGetGlyphs
01 usp10!ShlShape
02 usp10!ScriptShape
03 usp10!RenderItemNoFallback
04 usp10!RenderItemWithFallback
05 usp10!RenderItem
06 usp10!ScriptStringAnalyzeGlyphs
07 usp10!ScriptStringAnalyse
08 lpk!LpkCharsetDraw
09 lpk!LpkDrawTextEx
0a user32!DT_GetExtentMinusPrefixes
0b user32!NeedsEndEllipsis
0c user32!AddEllipsisAndDrawLine
0d user32!DrawTextExWorker
0e user32!DrawTextW
0f comctl32!CLVView::_ComputeLabelSizeWorker
10 comctl32!CLVView::v_RecomputeLabelSize
11 comctl32!CLVListView::v_DrawItem
12 comctl32!CLVDrawItemManager::DrawItem
13 comctl32!CLVDrawManager::_PaintItems
14 comctl32!CLVDrawManager::_PaintWorkArea
15 comctl32!CLVDrawManager::_OnPaintWorkAreas
16 comctl32!CLVDrawManager::_OnPaint
17 comctl32!CListView::WndProc
18 comctl32!CListView::s_WndProc
19 user32!UserCallWinProcCheckWow
1a user32!CallWindowProcAorW
1b user32!CallWindowProcW
1c comctl32!CallOriginalWndProc
1d comctl32!CallNextSubclassProc
1e comctl32!DefSubclassProc
1f shell32!DefSubclassProc
20 shell32!CListViewHost::s_ListViewSubclassWndProc
21 comctl32!CallNextSubclassProc
22 comctl32!MasterSubclassProc
23 user32!UserCallWinProcCheckWow
24 user32!DispatchClientMessage
25 user32!_fnDWORD
26 ntdll!KiUserCallbackDispatcherContinue
27 user32!NtUserDispatchMessage
28 user32!DispatchMessageWorker
29 user32!IsDialogMessageW
2a comctl32!Prop_IsDialogMessage
2b comctl32!_RealPropertySheet
2c comctl32!_PropertySheet
2d comdlg32!Print_InvokePropertySheets
2e comdlg32!PrintDlgExX
2f comdlg32!PrintDlgExW
30 notepad!GetPrinterDCviaDialog
31 notepad!PrintIt
32 notepad!NPCommand
33 notepad!NPWndProc
34 user32!UserCallWinProcCheckWow
35 user32!DispatchMessageWorker
36 notepad!WinMain
37 notepad!DisplayNonGenuineDlgWorker
38 kernel32!BaseThreadInitThunk
39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!NtWaitForSingleObject
01 ntdll!RtlpWaitOnCriticalSection
02 ntdll!RtlEnterCriticalSection
03 user32!BitmapFromDIB
04 user32!ConvertDIBBitmap
05 user32!ConvertDIBIcon
06 user32!CreateIconFromResourceEx
07 user32!PrivateEnumProc
08 kernel32!EnumResourceNamesInternal
09 kernel32!EnumResourceNamesExW
0a user32!PrivateExtractIconsW
0b shell32!SHPrivateExtractIcons
0c shell32!SHDefExtractIconW
0d shell32!CExtractIcon::_ExtractW
0e shell32!CExtractIconBase::Extract
0f shell32!IExtractIcon_Extract
10 shell32!_GetILIndexGivenPXIcon
11 shell32!_GetILIndexFromItem
12 shell32!SHGetIconIndexFromPIDL
13 shell32!MapIDListToIconILIndex
14 shell32!CLoadSystemIconTask::InternalResumeRT
15 shell32!CRunnableTask::Run
16 shell32!CShellTask::TT_Run
17 shell32!CShellTaskThread::ThreadProc
18 shell32!CShellTaskThread::s_ThreadProc
19 shlwapi!ExecuteWorkItemThreadProc
1a ntdll!RtlpTpWorkCallback
1b ntdll!TppWorkerThread
1c kernel32!BaseThreadInitThunk
1d ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!NtWaitForSingleObject
01 ntdll!RtlpWaitOnCriticalSection
02 ntdll!RtlEnterCriticalSection
03 user32!BitmapFromDIB
04 user32!ConvertDIBBitmap
05 user32!ConvertDIBIcon
06 user32!CreateIconFromResourceEx
07 user32!PrivateEnumProc
08 kernel32!EnumResourceNamesInternal
09 kernel32!EnumResourceNamesExW
0a user32!PrivateExtractIconsW
0b shell32!SHPrivateExtractIcons
0c shell32!SHDefExtractIconW
0d shell32!CExtractIcon::_ExtractW
0e shell32!CExtractIconBase::Extract
0f shell32!IExtractIcon_Extract
10 shell32!_GetILIndexGivenPXIcon
11 shell32!_GetILIndexFromItem
12 shell32!SHGetIconIndexFromPIDL
13 shell32!MapIDListToIconILIndex
14 shell32!CLoadSystemIconTask::InternalResumeRT
15 shell32!CRunnableTask::Run
16 shell32!CShellTask::TT_Run
17 shell32!CShellTaskThread::ThreadProc
18 shell32!CShellTaskThread::s_ThreadProc
19 shlwapi!ExecuteWorkItemThreadProc
1a ntdll!RtlpTpWorkCallback
1b ntdll!TppWorkerThread
1c kernel32!BaseThreadInitThunk
1d ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 user32!NtUserGetIconInfo
01 user32!CopyIcoCur
02 user32!InternalCopyImage
03 user32!CopyImage
04 comctl32!CImageList::_ReplaceIcon
05 comctl32!CImageList::ReplaceIcon
06 comctl32!CSparseImageList::ReplaceIcon
07 shell32!CIconCache::AddToBackIconTable
08 shell32!AddToBackIconTable
09 shell32!SHAddIconsToCache
0a shell32!_GetILIndexGivenPXIcon
0b shell32!_GetILIndexFromItem
0c shell32!SHGetIconIndexFromPIDL
0d shell32!MapIDListToIconILIndex
0e shell32!CLoadSystemIconTask::InternalResumeRT
0f shell32!CRunnableTask::Run
10 shell32!CShellTask::TT_Run
11 shell32!CShellTaskThread::ThreadProc
12 shell32!CShellTaskThread::s_ThreadProc
13 shlwapi!ExecuteWorkItemThreadProc
14 ntdll!RtlpTpWorkCallback
15 ntdll!TppWorkerThread
16 kernel32!BaseThreadInitThunk
17 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 gdi32!NtUserSelectPalette
01 gdi32!SelectPalette
02 gdi32!SetDIBits
03 user32!BitmapFromDIB
04 user32!ConvertDIBBitmap
05 user32!ConvertDIBIcon
06 user32!CreateIconFromResourceEx
07 user32!PrivateEnumProc
08 kernel32!EnumResourceNamesInternal
09 kernel32!EnumResourceNamesExW
0a user32!PrivateExtractIconsW
0b shell32!SHPrivateExtractIcons
0c shell32!SHDefExtractIconW
0d shell32!CExtractIcon::_ExtractW
0e shell32!CExtractIconBase::Extract
0f shell32!IExtractIcon_Extract
10 shell32!_GetILIndexGivenPXIcon
11 shell32!_GetILIndexFromItem
12 shell32!SHGetIconIndexFromPIDL
13 shell32!MapIDListToIconILIndex
14 shell32!CLoadSystemIconTask::InternalResumeRT
15 shell32!CRunnableTask::Run
16 shell32!CShellTask::TT_Run
17 shell32!CShellTaskThread::ThreadProc
18 shell32!CShellTaskThread::s_ThreadProc
19 shlwapi!ExecuteWorkItemThreadProc
1a ntdll!RtlpTpWorkCallback
1b ntdll!TppWorkerThread
1c kernel32!BaseThreadInitThunk
1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjectsExImplementation
03 user32!RealMsgWaitForMultipleObjectsEx
04 user32!MsgWaitForMultipleObjectsEx
05 user32!MsgWaitForMultipleObjects
06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects
07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating
08 shell32!CShellTaskThread::ThreadProc
09 shell32!CShellTaskThread::s_ThreadProc
0a shlwapi!ExecuteWorkItemThreadProc
0b ntdll!RtlpTpWorkCallback
0c ntdll!TppWorkerThread
0d kernel32!BaseThreadInitThunk
0e ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
# Call Site
00 ntdll!ZwDelayExecution
01 KERNELBASE!SleepEx
02 ole32!CROIDTable::WorkerThreadLoop
03 ole32!CRpcThread::WorkerLoop
04 ole32!CRpcThreadCache::RpcWorkerThreadEntry
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CCALL::SendReceive
02 rpcrt4!NdrpClientCall3
03 rpcrt4!NdrClientCall3
04 winspool!RpcFindNextPrinterChangeNotification
05 winspool!FindNextPrinterChangeNotification
06 prncache!PrintCache::Listeners::Listener::ProcessWait
07 prncache!PrintCache::Listeners::Listener::ProcessWaitCB
08 ntdll!TppWaitpExecuteCallback
09 ntdll!TppWorkerThread
0a kernel32!BaseThreadInitThunk
0b ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

// 12th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen
# Call Site
00 usp10!otlChainingLookup::apply
01 usp10!ApplyLookup
02 usp10!ApplyFeatures
03 usp10!SubstituteOtlGlyphs
04 usp10!GenericEngineGetGlyphs
05 usp10!ShlShape
06 usp10!ScriptShape
07 usp10!RenderItemNoFallback
08 usp10!RenderItemWithFallback
09 usp10!RenderItem
0a usp10!ScriptStringAnalyzeGlyphs
0b usp10!ScriptStringAnalyse
0c lpk!LpkCharsetDraw
0d lpk!LpkDrawTextEx
0e user32!DT_DrawStr
0f user32!DT_DrawJustifiedLine
10 user32!DrawTextExWorker
11 user32!DrawTextW
12 comctl32!CLVView::_ComputeLabelSizeWorker
13 comctl32!CLVView::v_RecomputeLabelSize
14 comctl32!CLVListView::v_DrawItem
15 comctl32!CLVDrawItemManager::DrawItem
16 comctl32!CLVDrawManager::_PaintItems
17 comctl32!CLVDrawManager::_PaintWorkArea
18 comctl32!CLVDrawManager::_OnPaintWorkAreas
19 comctl32!CLVDrawManager::_OnPaint
1a comctl32!CListView::WndProc
1b comctl32!CListView::s_WndProc
1c user32!UserCallWinProcCheckWow
1d user32!CallWindowProcAorW
1e user32!CallWindowProcW
1f comctl32!CallOriginalWndProc
20 comctl32!CallNextSubclassProc
21 comctl32!DefSubclassProc
22 shell32!DefSubclassProc
23 shell32!CListViewHost::s_ListViewSubclassWndProc
24 comctl32!CallNextSubclassProc
25 comctl32!MasterSubclassProc
26 user32!UserCallWinProcCheckWow
27 user32!DispatchClientMessage
28 user32!_fnDWORD
29 ntdll!KiUserCallbackDispatcherContinue
2a user32!NtUserDispatchMessage
2b user32!DispatchMessageWorker
2c user32!IsDialogMessageW
2d comctl32!Prop_IsDialogMessage
2e comctl32!_RealPropertySheet
2f comctl32!_PropertySheet
30 comdlg32!Print_InvokePropertySheets
31 comdlg32!PrintDlgExX
32 comdlg32!PrintDlgExW
33 notepad!GetPrinterDCviaDialog
34 notepad!PrintIt
35 notepad!NPCommand
36 notepad!NPWndProc
37 user32!UserCallWinProcCheckWow
38 user32!DispatchMessageWorker
39 notepad!WinMain
3a notepad!DisplayNonGenuineDlgWorker
3b kernel32!BaseThreadInitThunk
3c ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!TppWaiterpThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
# Call Site
00 ntdll!NtUnmapViewOfSection
01 KERNELBASE!FreeLibrary
02 user32!PrivateExtractIconsW
03 shell32!SHPrivateExtractIcons
04 shell32!SHDefExtractIconW
05 shell32!CExtractIcon::_ExtractW
06 shell32!CExtractIconBase::Extract
07 shell32!IExtractIcon_Extract
08 shell32!_GetILIndexGivenPXIcon
09 shell32!_GetILIndexFromItem
0a shell32!SHGetIconIndexFromPIDL
0b shell32!MapIDListToIconILIndex
0c shell32!CLoadSystemIconTask::InternalResumeRT
0d shell32!CRunnableTask::Run
0e shell32!CShellTask::TT_Run
0f shell32!CShellTaskThread::ThreadProc
10 shell32!CShellTaskThread::s_ThreadProc
11 shlwapi!ExecuteWorkItemThreadProc
12 ntdll!RtlpTpWorkCallback
13 ntdll!TppWorkerThread
14 kernel32!BaseThreadInitThunk
15 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjects
03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges
04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc
05 kernel32!BaseThreadInitThunk
06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjectsExImplementation
03 user32!RealMsgWaitForMultipleObjectsEx
04 user32!MsgWaitForMultipleObjectsEx
05 user32!MsgWaitForMultipleObjects
06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects
07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating
08 shell32!CShellTaskThread::ThreadProc
09 shell32!CShellTaskThread::s_ThreadProc
0a shlwapi!ExecuteWorkItemThreadProc
0b ntdll!RtlpTpWorkCallback
0c ntdll!TppWorkerThread
0d kernel32!BaseThreadInitThunk
0e ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 kernel32!WaitForMultipleObjectsExImplementation
03 user32!RealMsgWaitForMultipleObjectsEx
04 user32!MsgWaitForMultipleObjectsEx
05 user32!MsgWaitForMultipleObjects
06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects
07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating
08 shell32!CShellTaskThread::ThreadProc
09 shell32!CShellTaskThread::s_ThreadProc
0a shlwapi!ExecuteWorkItemThreadProc
0b ntdll!RtlpTpWorkCallback
0c ntdll!TppWorkerThread
0d kernel32!BaseThreadInitThunk
0e ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
# Call Site
00 gdi32!ZwGdiSetDIBitsToDeviceInternal
01 gdi32!SetDIBitsToDevice
02 gdi32!SetDIBits
03 user32!BitmapFromDIB
04 user32!ConvertDIBBitmap
05 user32!ConvertDIBIcon
06 user32!CreateIconFromResourceEx
07 user32!PrivateEnumProc
08 kernel32!EnumResourceNamesInternal
09 kernel32!EnumResourceNamesExW
0a user32!PrivateExtractIconsW
0b shell32!SHPrivateExtractIcons
0c shell32!SHDefExtractIconW
0d shell32!CExtractIcon::_ExtractW
0e shell32!CExtractIconBase::Extract
0f shell32!IExtractIcon_Extract
10 shell32!_GetILIndexGivenPXIcon
11 shell32!_GetILIndexFromItem
12 shell32!SHGetIconIndexFromPIDL
13 shell32!MapIDListToIconILIndex
14 shell32!CLoadSystemIconTask::InternalResumeRT
15 shell32!CRunnableTask::Run
16 shell32!CShellTask::TT_Run
17 shell32!CShellTaskThread::ThreadProc
18 shell32!CShellTaskThread::s_ThreadProc
19 shlwapi!ExecuteWorkItemThreadProc
1a ntdll!RtlpTpWorkCallback
1b ntdll!TppWorkerThread
1c kernel32!BaseThreadInitThunk
1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
# Call Site
00 ntdll!ZwMapViewOfSection
01 KERNELBASE!BasepLoadLibraryAsDataFileInternal
02 KERNELBASE!LoadLibraryExW
03 user32!PrivateExtractIconsW
04 shell32!SHPrivateExtractIcons
05 shell32!SHDefExtractIconW
06 shell32!CExtractIcon::_ExtractW
07 shell32!CExtractIconBase::Extract
08 shell32!IExtractIcon_Extract
09 shell32!_GetILIndexGivenPXIcon
0a shell32!_GetILIndexFromItem
0b shell32!SHGetIconIndexFromPIDL
0c shell32!MapIDListToIconILIndex
0d shell32!CLoadSystemIconTask::InternalResumeRT
0e shell32!CRunnableTask::Run
0f shell32!CShellTask::TT_Run
10 shell32!CShellTaskThread::ThreadProc
11 shell32!CShellTaskThread::s_ThreadProc
12 shlwapi!ExecuteWorkItemThreadProc
13 ntdll!RtlpTpWorkCallback
14 ntdll!TppWorkerThread
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
# Call Site
00 ntdll!ZwAlpcSendWaitReceivePort
01 rpcrt4!LRPC_CASSOCIATION::AlpcSendWaitReceivePort
02 rpcrt4!LRPC_BASE_CCALL::DoSendReceive
03 rpcrt4!LRPC_BASE_CCALL::SendReceive
04 rpcrt4!NdrpClientCall2
05 rpcrt4!NdrClientCall2
06 ole32!CRpcResolver::BulkUpdateOIDs
07 ole32!CROIDTable::ClientBulkUpdateOIDWithPingServer
08 ole32!CROIDTable::WorkerThreadLoop
09 ole32!CRpcThread::WorkerLoop
0a ole32!CRpcThreadCache::RpcWorkerThreadEntry
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 kernel32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

11 Id: 1344.1fb0 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
# Call Site
00 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Spiking Thread, Top Module, Module Hint, and Memory Fluctuation: pattern cooperation

Saturday, October 24th, 2015

We noticed that after restarting Windows 7 system on one of our notebooks it becomes very sluggish. Task Manager showed 25% CPU usage in one of svchost.exe processes and very high usage of physical memory. So we immediately dumped it using procdump. The resulted process memory dump was almost 1.5 GB. Although the analysis case is very simple and straightforward we decided to publish to show the value of crash and hang dump analysis in understanding abnormal software behavior in “user” context.

When we open the memory dump and run !runaway WinDbg command we immediately recognize Spiking Thread pattern:

0:000> !runaway f
User Mode Time
Thread Time
38:14a0 0 days 0:04:59.911
36:a88 0 days 0:00:00.187
2:41c 0 days 0:00:00.187
39:19f8 0 days 0:00:00.046
34:fa8 0 days 0:00:00.046
21:12b4 0 days 0:00:00.031
43:8f0 0 days 0:00:00.015
42:1504 0 days 0:00:00.015
41:1a20 0 days 0:00:00.015
40:978 0 days 0:00:00.015
33:e0c 0 days 0:00:00.015
32:ff8 0 days 0:00:00.015
22:1304 0 days 0:00:00.015
19:f68 0 days 0:00:00.015
9:664 0 days 0:00:00.015
8:660 0 days 0:00:00.015
6:518 0 days 0:00:00.015
4:4a4 0 days 0:00:00.015
51:160c 0 days 0:00:00.000
50:1590 0 days 0:00:00.000
49:15d8 0 days 0:00:00.000
48:ac8 0 days 0:00:00.000
47:14d0 0 days 0:00:00.000
46:1bfc 0 days 0:00:00.000
45:18e8 0 days 0:00:00.000
44:1448 0 days 0:00:00.000
37:1910 0 days 0:00:00.000
35:558 0 days 0:00:00.000
31:14b8 0 days 0:00:00.000
30:14b4 0 days 0:00:00.000
29:14ac 0 days 0:00:00.000
28:13d0 0 days 0:00:00.000
27:13c8 0 days 0:00:00.000
26:13b4 0 days 0:00:00.000
25:13b0 0 days 0:00:00.000
24:13a8 0 days 0:00:00.000
23:1328 0 days 0:00:00.000
20:12b0 0 days 0:00:00.000
18:e20 0 days 0:00:00.000
17:e10 0 days 0:00:00.000
16:dd0 0 days 0:00:00.000
15:ce0 0 days 0:00:00.000
14:754 0 days 0:00:00.000
13:718 0 days 0:00:00.000
12:678 0 days 0:00:00.000
11:674 0 days 0:00:00.000
10:668 0 days 0:00:00.000
7:548 0 days 0:00:00.000
5:4ac 0 days 0:00:00.000
3:4a0 0 days 0:00:00.000
1:418 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Kernel Mode Time
Thread Time
38:14a0 0 days 0:00:55.707
36:a88 0 days 0:00:01.778
2:41c 0 days 0:00:00.405
34:fa8 0 days 0:00:00.109
9:664 0 days 0:00:00.062
43:8f0 0 days 0:00:00.046
42:1504 0 days 0:00:00.046
21:12b4 0 days 0:00:00.046
32:ff8 0 days 0:00:00.031
22:1304 0 days 0:00:00.031
18:e20 0 days 0:00:00.031
39:19f8 0 days 0:00:00.015
19:f68 0 days 0:00:00.015
6:518 0 days 0:00:00.015
3:4a0 0 days 0:00:00.015
1:418 0 days 0:00:00.015
51:160c 0 days 0:00:00.000
50:1590 0 days 0:00:00.000
49:15d8 0 days 0:00:00.000
48:ac8 0 days 0:00:00.000
47:14d0 0 days 0:00:00.000
46:1bfc 0 days 0:00:00.000
45:18e8 0 days 0:00:00.000
44:1448 0 days 0:00:00.000
41:1a20 0 days 0:00:00.000
40:978 0 days 0:00:00.000
37:1910 0 days 0:00:00.000
35:558 0 days 0:00:00.000
33:e0c 0 days 0:00:00.000
31:14b8 0 days 0:00:00.000
30:14b4 0 days 0:00:00.000
29:14ac 0 days 0:00:00.000
28:13d0 0 days 0:00:00.000
27:13c8 0 days 0:00:00.000
26:13b4 0 days 0:00:00.000
25:13b0 0 days 0:00:00.000
24:13a8 0 days 0:00:00.000
23:1328 0 days 0:00:00.000
20:12b0 0 days 0:00:00.000
17:e10 0 days 0:00:00.000
16:dd0 0 days 0:00:00.000
15:ce0 0 days 0:00:00.000
14:754 0 days 0:00:00.000
13:718 0 days 0:00:00.000
12:678 0 days 0:00:00.000
11:674 0 days 0:00:00.000
10:668 0 days 0:00:00.000
8:660 0 days 0:00:00.000
7:548 0 days 0:00:00.000
5:4ac 0 days 0:00:00.000
4:4a4 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Elapsed Time
Thread Time
0:410 0 days 0:10:24.550
2:41c 0 days 0:10:24.534
1:418 0 days 0:10:24.534
4:4a4 0 days 0:10:24.331
3:4a0 0 days 0:10:24.331
5:4ac 0 days 0:10:24.269
6:518 0 days 0:10:23.957
7:548 0 days 0:10:23.817
8:660 0 days 0:10:22.176
9:664 0 days 0:10:22.156
10:668 0 days 0:10:22.126
11:674 0 days 0:10:22.026
12:678 0 days 0:10:21.986
13:718 0 days 0:10:20.066
14:754 0 days 0:10:20.056
15:ce0 0 days 0:10:15.131
16:dd0 0 days 0:10:14.641
17:e10 0 days 0:10:14.551
18:e20 0 days 0:10:14.531
19:f68 0 days 0:10:13.611
21:12b4 0 days 0:10:10.647
20:12b0 0 days 0:10:10.647
22:1304 0 days 0:10:10.553
23:1328 0 days 0:10:10.381
24:13a8 0 days 0:10:09.024
26:13b4 0 days 0:10:08.931
25:13b0 0 days 0:10:08.931
28:13d0 0 days 0:10:08.899
27:13c8 0 days 0:10:08.899
31:14b8 0 days 0:10:07.932
30:14b4 0 days 0:10:07.932
29:14ac 0 days 0:10:07.932
32:ff8 0 days 0:08:11.785
33:e0c 0 days 0:08:11.644
34:fa8 0 days 0:08:06.750
35:558 0 days 0:08:05.765
36:a88 0 days 0:08:05.127
37:1910 0 days 0:08:02.608
38:14a0 0 days 0:07:19.276
42:1504 0 days 0:04:55.634
41:1a20 0 days 0:04:55.634
40:978 0 days 0:04:55.634
39:19f8 0 days 0:04:55.634
43:8f0 0 days 0:04:55.618
44:1448 0 days 0:04:42.634
46:1bfc 0 days 0:04:20.945
45:18e8 0 days 0:04:20.945
47:14d0 0 days 0:02:42.515
48:ac8 0 days 0:01:27.434
50:1590 0 days 0:00:04.917
49:15d8 0 days 0:00:04.917
51:160c 0 days 0:00:03.387

We see that most of the time the thread was consuming CPU in user mode and less in kernel mode.

We then examine that thread stack trace:

0:000> ~38k
# Child-SP RetAddr Call Site
00 00000000`0414ddd8 00000000`7769cf66 ntdll!ZwQueryPerformanceCounter+0xa
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll -
01 00000000`0414dde0 000007fe`ee7ee7aa ntdll!RtlQueryPerformanceFrequency+0x16
02 00000000`0414de10 000007fe`ee7ee53f wuaueng!DllInstall+0×153da
03 00000000`0414de40 000007fe`ee7e0d99 wuaueng!DllInstall+0×1516f
04 00000000`0414e190 000007fe`ee7df542 wuaueng!DllInstall+0×79c9
05 00000000`0414e1f0 000007fe`ee7df57c wuaueng!DllInstall+0×6172
06 00000000`0414e2b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
07 00000000`0414e370 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
08 00000000`0414e430 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
09 00000000`0414e4f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0a 00000000`0414e5b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0b 00000000`0414e670 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0c 00000000`0414e730 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0d 00000000`0414e7f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0e 00000000`0414e8b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0f 00000000`0414e970 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
10 00000000`0414ea30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
11 00000000`0414eaf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
12 00000000`0414ebb0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
13 00000000`0414ec70 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
14 00000000`0414ed30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
15 00000000`0414edf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
16 00000000`0414eeb0 000007fe`ee7df2d7 wuaueng!DllInstall+0×61ac
17 00000000`0414ef70 000007fe`ee7e9b4f wuaueng!DllInstall+0×5f07
18 00000000`0414f020 000007fe`ee7eb7e8 wuaueng!DllInstall+0×1077f
19 00000000`0414f370 000007fe`ee8010b2 wuaueng!DllInstall+0×12418
1a 00000000`0414f5b0 000007fe`ee7fe53e wuaueng!DllInstall+0×27ce2
1b 00000000`0414f610 000007fe`ee7fccac wuaueng!DllInstall+0×2516e
1c 00000000`0414f660 000007fe`ee7dec19 wuaueng!DllInstall+0×238dc
1d 00000000`0414f690 000007fe`ee7de30f wuaueng!DllInstall+0×5849
1e 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0×4f3f
1f 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd
20 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that Top Module is from Windows Update Agent:

0:000> lmv m wuaueng
Browse full module list
start end module name
000007fe`ee740000 000007fe`ee9a0000 wuaueng (export symbols) wuaueng.dll
Loaded symbol image file: wuaueng.dll
Image path: c:\Windows\System32\wuaueng.dll
Image name: wuaueng.dll
Browse all global symbols functions data
Timestamp: Wed May 14 17:21:24 2014 (53739804)
CheckSum: 00265DEA
ImageSize: 00260000
File version: 7.6.7600.320
Product version: 7.6.7600.320
File flags: 8 (Mask 3F) Private
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: wuaueng.dll
OriginalFilename: wuaueng.dll
ProductVersion: 7.6.7600.320
FileVersion: 7.6.7600.320 (winmain_wtr_wsus3sp2(oobla).140514-0916)
FileDescription: Windows Update Agent
LegalCopyright: © Microsoft Corporation. All rights reserved.

We also examine process address space:

0:000> !address -summary

Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 507 7ff`9373b000 ( 7.998 TB) 99.98%
Heap 574 0`3e2ad000 ( 994.676 MB) 57.27% 0.01%
9786 0`2803e000 ( 640.242 MB) 36.87% 0.01%
Image 874 0`049a6000 ( 73.648 MB) 4.24% 0.00%
Stack 156 0`01a00000 ( 26.000 MB) 1.50% 0.00%
Other 13 0`001bb000 ( 1.730 MB) 0.10% 0.00%
TEB 52 0`00068000 ( 416.000 kB) 0.02% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 834 0`5345e000 ( 1.301 GB) 76.72% 0.02%
MEM_MAPPED 9747 0`14ab0000 ( 330.688 MB) 19.04% 0.00%
MEM_IMAGE 875 0`049a7000 ( 73.652 MB) 4.24% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 507 7ff`9373b000 ( 7.998 TB) 99.98%
MEM_COMMIT 11164 0`513d8000 ( 1.269 GB) 74.85% 0.02%
MEM_RESERVE 292 0`1b4dd000 ( 436.863 MB) 25.15% 0.01%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 5533 0`41d6c000 ( 1.029 GB) 60.66% 0.01%
PAGE_WRITECOPY 4911 0`098b9000 ( 152.723 MB) 8.79% 0.00%
PAGE_READONLY 504 0`03204000 ( 50.016 MB) 2.88% 0.00%
PAGE_EXECUTE_READ 164 0`02b29000 ( 43.160 MB) 2.49% 0.00%
PAGE_READWRITE|PAGE_GUARD 52 0`00086000 ( 536.000 kB) 0.03% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0`ff84b000 7fd`eed65000 ( 7.992 TB)
Heap 0`83eb0000 0`01c17000 ( 28.090 MB)
0`93070000 0`0ffbd000 ( 255.738 MB)
Image 7fe`fe18a000 0`0089e000 ( 8.617 MB)
Stack 0`07140000 0`0007b000 ( 492.000 kB)
Other 0`00760000 0`00181000 ( 1.504 MB)
TEB 7ff`fff0a000 0`00002000 ( 8.000 kB)
PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

We see that process heap occupies almost 1 GB. Let’s see its statistics:

0:000> !heap -s

******************************
NT HEAP STATS BELOW
******************************
LFH Key : 0x000000a57ddeb5ed
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
Virtual block: 0000000051eb0000 - 0000000051eb0000 (size 0000000000000000)
Virtual block: 0000000052930000 - 0000000052930000 (size 0000000000000000)
Virtual block: 0000000026ba0000 - 0000000026ba0000 (size 0000000000000000)
Virtual block: 0000000053e50000 - 0000000053e50000 (size 0000000000000000)
Virtual block: 0000000054490000 - 0000000054490000 (size 0000000000000000)
Virtual block: 00000000547a0000 - 00000000547a0000 (size 0000000000000000)
Virtual block: 000000003bda0000 - 000000003bda0000 (size 0000000000000000)
Virtual block: 0000000056030000 - 0000000056030000 (size 0000000000000000)
Virtual block: 00000000567c0000 - 00000000567c0000 (size 0000000000000000)
Virtual block: 00000000572a0000 - 00000000572a0000 (size 0000000000000000)
Virtual block: 0000000057870000 - 0000000057870000 (size 0000000000000000)
Virtual block: 0000000045a10000 - 0000000045a10000 (size 0000000000000000)
Virtual block: 0000000058fb0000 - 0000000058fb0000 (size 0000000000000000)
Virtual block: 0000000045c10000 - 0000000045c10000 (size 0000000000000000)
Virtual block: 00000000599c0000 - 00000000599c0000 (size 0000000000000000)
Virtual block: 0000000059ff0000 - 0000000059ff0000 (size 0000000000000000)
Virtual block: 000000005ae20000 - 000000005ae20000 (size 0000000000000000)
Virtual block: 000000005c5d0000 - 000000005c5d0000 (size 0000000000000000)
Virtual block: 0000000054b90000 - 0000000054b90000 (size 0000000000000000)
Virtual block: 000000005d070000 - 000000005d070000 (size 0000000000000000)
Virtual block: 000000006e370000 - 000000006e370000 (size 0000000000000000)
Virtual block: 000000006f8e0000 - 000000006f8e0000 (size 0000000000000000)
Virtual block: 000000006ed20000 - 000000006ed20000 (size 0000000000000000)
Virtual block: 0000000070890000 - 0000000070890000 (size 0000000000000000)
Virtual block: 000000005e370000 - 000000005e370000 (size 0000000000000000)
Virtual block: 000000005f5a0000 - 000000005f5a0000 (size 0000000000000000)
Virtual block: 000000005fa60000 - 000000005fa60000 (size 0000000000000000)
Virtual block: 000000005ffe0000 - 000000005ffe0000 (size 0000000000000000)
Virtual block: 0000000060770000 - 0000000060770000 (size 0000000000000000)
Virtual block: 0000000060aa0000 - 0000000060aa0000 (size 0000000000000000)
Virtual block: 0000000061810000 - 0000000061810000 (size 0000000000000000)
Virtual block: 0000000061a30000 - 0000000061a30000 (size 0000000000000000)
Virtual block: 0000000061f00000 - 0000000061f00000 (size 0000000000000000)
Virtual block: 0000000064470000 - 0000000064470000 (size 0000000000000000)
Virtual block: 0000000064c00000 - 0000000064c00000 (size 0000000000000000)
Virtual block: 00000000656d0000 - 00000000656d0000 (size 0000000000000000)
Virtual block: 00000000660f0000 - 00000000660f0000 (size 0000000000000000)
Virtual block: 0000000066530000 - 0000000066530000 (size 0000000000000000)
Virtual block: 00000000669d0000 - 00000000669d0000 (size 0000000000000000)
Virtual block: 00000000676a0000 - 00000000676a0000 (size 0000000000000000)
Virtual block: 0000000067a70000 - 0000000067a70000 (size 0000000000000000)
Virtual block: 0000000068a20000 - 0000000068a20000 (size 0000000000000000)
Virtual block: 0000000069f10000 - 0000000069f10000 (size 0000000000000000)
Virtual block: 000000006a6c0000 - 000000006a6c0000 (size 0000000000000000)
Virtual block: 000000006ad80000 - 000000006ad80000 (size 0000000000000000)
Virtual block: 000000006b9a0000 - 000000006b9a0000 (size 0000000000000000)
Virtual block: 000000006bb40000 - 000000006bb40000 (size 0000000000000000)
Virtual block: 000000006c4f0000 - 000000006c4f0000 (size 0000000000000000)
Virtual block: 000000006dc30000 - 000000006dc30000 (size 0000000000000000)
Virtual block: 000000006de10000 - 000000006de10000 (size 0000000000000000)
Virtual block: 000000006ef80000 - 000000006ef80000 (size 0000000000000000)
Virtual block: 00000000728f0000 - 00000000728f0000 (size 0000000000000000)
Virtual block: 0000000071270000 - 0000000071270000 (size 0000000000000000)
Virtual block: 0000000074030000 - 0000000074030000 (size 0000000000000000)
Virtual block: 00000000746c0000 - 00000000746c0000 (size 0000000000000000)
Virtual block: 00000000749d0000 - 00000000749d0000 (size 0000000000000000)
Virtual block: 0000000074e50000 - 0000000074e50000 (size 0000000000000000)
Virtual block: 0000000071730000 - 0000000071730000 (size 0000000000000000)
Virtual block: 0000000075c30000 - 0000000075c30000 (size 0000000000000000)
Virtual block: 0000000075e00000 - 0000000075e00000 (size 0000000000000000)
Virtual block: 0000000075fe0000 - 0000000075fe0000 (size 0000000000000000)
Virtual block: 00000000761a0000 - 00000000761a0000 (size 0000000000000000)
Virtual block: 0000000076460000 - 0000000076460000 (size 0000000000000000)
Virtual block: 0000000076620000 - 0000000076620000 (size 0000000000000000)
Virtual block: 0000000077850000 - 0000000077850000 (size 0000000000000000)
Virtual block: 0000000076c60000 - 0000000076c60000 (size 0000000000000000)
Virtual block: 0000000078ec0000 - 0000000078ec0000 (size 0000000000000000)
Virtual block: 0000000062b30000 - 0000000062b30000 (size 0000000000000000)
Virtual block: 000000007a520000 - 000000007a520000 (size 0000000000000000)
Virtual block: 000000007ab50000 - 000000007ab50000 (size 0000000000000000)
Virtual block: 00000000770e0000 - 00000000770e0000 (size 0000000000000000)
Virtual block: 000000007af00000 - 000000007af00000 (size 0000000000000000)
Virtual block: 000000007c6b0000 - 000000007c6b0000 (size 0000000000000000)
Virtual block: 000000007de50000 - 000000007de50000 (size 0000000000000000)
Virtual block: 000000007e2d0000 - 000000007e2d0000 (size 0000000000000000)
Virtual block: 000000007fff0000 - 000000007fff0000 (size 0000000000000000)
Virtual block: 00000000628c0000 - 00000000628c0000 (size 0000000000000000)
Virtual block: 00000000809b0000 - 00000000809b0000 (size 0000000000000000)
Virtual block: 0000000080d80000 - 0000000080d80000 (size 0000000000000000)
Virtual block: 0000000081310000 - 0000000081310000 (size 0000000000000000)
Virtual block: 00000000631a0000 - 00000000631a0000 (size 0000000000000000)
Virtual block: 0000000062c70000 - 0000000062c70000 (size 0000000000000000)
Virtual block: 0000000092ca0000 - 0000000092ca0000 (size 0000000000000000)
Virtual block: 0000000003940000 - 0000000003940000 (size 0000000000000000)
Virtual block: 0000000081cd0000 - 0000000081cd0000 (size 0000000000000000)
Virtual block: 0000000082260000 - 0000000082260000 (size 0000000000000000)
Virtual block: 0000000077200000 - 0000000077200000 (size 0000000000000000)
Virtual block: 000000007ec90000 - 000000007ec90000 (size 0000000000000000)
Virtual block: 000000007edb0000 - 000000007edb0000 (size 0000000000000000)
Virtual block: 000000007eec0000 - 000000007eec0000 (size 0000000000000000)
Virtual block: 00000000829f0000 - 00000000829f0000 (size 0000000000000000)
Virtual block: 0000000003770000 - 0000000003770000 (size 0000000000000000)
Virtual block: 0000000082fc0000 - 0000000082fc0000 (size 0000000000000000)
Virtual block: 0000000083210000 - 0000000083210000 (size 0000000000000000)
Virtual block: 0000000083320000 - 0000000083320000 (size 0000000000000000)
Virtual block: 0000000083760000 - 0000000083760000 (size 0000000000000000)
Virtual block: 0000000083870000 - 0000000083870000 (size 0000000000000000)
Virtual block: 0000000083eb0000 - 0000000083eb0000 (size 0000000000000000)
Virtual block: 0000000085ad0000 - 0000000085ad0000 (size 0000000000000000)
Virtual block: 0000000086290000 - 0000000086290000 (size 0000000000000000)

0000000000340000 00000002 210688 174796 210688 60711 775 45 100 1 LFH
External fragmentation 34 % (775 free blocks)
0000000000010000 00008000 64 4 64 1 1 1 0 0
00000000005c0000 00001002 3136 1256 3136 278 39 3 0 0 LFH
External fragmentation 22 % (39 free blocks)
0000000000960000 00001002 512 8 512 2 1 1 0 0
0000000001520000 00001002 512 276 512 2 10 1 0 0 LFH
00000000013f0000 00001002 512 8 512 2 1 1 0 0
0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH
00000000010b0000 00001002 512 8 512 2 1 1 0 0
00000000019b0000 00001002 512 308 512 5 8 1 0 0 LFH
0000000001220000 00001002 512 380 512 63 21 1 0 0 LFH
00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH
0000000001900000 00001002 512 156 512 133 1 1 0 0
00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH
0000000002e00000 00001002 48256 16656 48256 9836 98 13 0 1 LFH
External fragmentation 59 % (98 free blocks)
0000000002f10000 00001002 3584 3280 3584 1773 79 3 0 5 LFH
External fragmentation 54 % (79 free blocks)
00000000035c0000 00001002 64 8 64 3 1 1 0 0
0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH
00000000038c0000 00001002 512 8 512 3 1 1 0 0
0000000003ad0000 00001002 512 8 512 3 1 1 0 0
0000000003d40000 00001002 512 8 512 3 1 1 0 0
00000000005b0000 00001002 64 8 64 3 1 1 0 0
0000000004280000 00011002 512 64 512 59 2 1 0 0
00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH
0000000004270000 00001002 1088 304 1088 21 5 2 0 0 LFH
0000000006650000 00000002 1088 80 1088 10 2 2 0 0
0000000007050000 00001002 512 8 512 3 1 1 0 0
0000000007cc0000 00001002 512 8 512 3 1 1 0 0
0000000000be0000 00001002 64 24 64 17 2 1 0 0
0000000006540000 00001002 1536 544 1536 259 4 2 0 0 LFH
0000000000b40000 00001002 64 8 64 3 1 1 0 0
0000000004950000 00001002 512 8 512 3 1 1 0 0
————————————————————————————-

We see many large virtual blocks allocated with less than 200 KB of normal blocks committed. We suspect possible Memory Leak. Let’s check a few such virtual blocks:

0:000> !address 0000000062b30000

Usage: Heap
Base Address: 00000000`62b30000
End Address: 00000000`62c65000
Region Size: 00000000`00135000 ( 1.207 MB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000000`62b30000
Allocation Protect: 00000004 PAGE_READWRITE
More info: heap owning the address: !heap 0×340000
More info: heap large/virtual block
More info: heap entry containing the address: !heap -x 0×62b30000

Content source: 1 (target), length: 135000

0:000> !address 0000000003770000

Usage: Heap
Base Address: 00000000`03770000
End Address: 00000000`03870000
Region Size: 00000000`00100000 ( 1.000 MB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000000`03770000
Allocation Protect: 00000004 PAGE_READWRITE
More info: heap owning the address: !heap 0×340000
More info: heap large/virtual block
More info: heap entry containing the address: !heap -x 0×3770000

Content source: 1 (target), length: 100000

0:000> !address 0000000085ad0000

Usage: Heap
Base Address: 00000000`85ad0000
End Address: 00000000`8628d000
Region Size: 00000000`007bd000 ( 7.738 MB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000000`85ad0000
Allocation Protect: 00000004 PAGE_READWRITE
More info: heap owning the address: !heap 0×340000
More info: heap large/virtual block
More info: heap entry containing the address: !heap -x 0×85ad0000

Content source: 1 (target), length: 7bd000

We see they all contain assembly packages for KB articles for various updates (Module Hints):

0:000> dc 00000000`85ad0000
00000000`85ad0000 86290000 00000000 83eb0000 00000000 ..)………….
00000000`85ad0010 00000000 00000000 00000000 00000000 …………….
00000000`85ad0020 007bd000 00000000 007bd000 00000000 ..{…….{…..
00000000`85ad0030 00000000 00000000 f333297f 04000000 ………)3…..
00000000`85ad0040 00000001 007bc51a 00000004 00000000 ……{………
00000000`85ad0050 0061003c 00730073 006d0065 006c0062 <.a.s.s.e.m.b.l.
00000000`85ad0060 00200079 006d0078 006e006c 003d0073 y. .x.m.l.n.s.=.
00000000`85ad0070 00750022 006e0072 0073003a 00680063 “.u.r.n.:.s.c.h.

0:000> du 00000000`85ad0050
00000000`85ad0050 “<assembly xmlns=”urn:schemas-mic”
00000000`85ad0090 “rosoft-com:asm.v3″ manifestVersi”
00000000`85ad00d0 “on=”1.0″ description=”Fix for KB”
00000000`85ad0110 “2798162″ displayName=”default” c”
00000000`85ad0150 “ompany=”Microsoft Corporation” c”
00000000`85ad0190 “opyright=”Microsoft Corporation”"
00000000`85ad01d0 ” supportInformation=”http://supp”
00000000`85ad0210 “ort.microsoft.com/?kbid=2798162″”
00000000`85ad0250 ” creationTimeStamp=”2013-04-15T0″
00000000`85ad0290 “4:10:39Z” lastUpdateTimeStamp=”2″
00000000`85ad02d0 “013-04-15T04:10:39Z”><assemblyid”
00000000`85ad0310 “entity name=”Package_for_KB27981″

However, in less than 20 minutes CPU and memory consumption normalized and the new saved process memory dump was less than 150 KB. We open it and see that CPU consumption happened for about 15 minutes (in the past) with the consuming thread now #32 instead of #38:

0:000> !runaway f
User Mode Time
Thread Time
32:14a0 0 days 0:13:24.029
33:b6c 0 days 0:00:00.078
41:174c 0 days 0:00:00.062
40:1bd8 0 days 0:00:00.062
30:fa8 0 days 0:00:00.046
45:650 0 days 0:00:00.015
42:4ac 0 days 0:00:00.015
39:990 0 days 0:00:00.015
29:e0c 0 days 0:00:00.015
28:ff8 0 days 0:00:00.015
18:1304 0 days 0:00:00.015
17:f68 0 days 0:00:00.015
7:664 0 days 0:00:00.015
6:660 0 days 0:00:00.015
4:518 0 days 0:00:00.015
3:4a4 0 days 0:00:00.015
47:9c8 0 days 0:00:00.000
46:f20 0 days 0:00:00.000
44:1440 0 days 0:00:00.000
43:11d0 0 days 0:00:00.000
38:db4 0 days 0:00:00.000
37:6ac 0 days 0:00:00.000
36:c4c 0 days 0:00:00.000
35:ff4 0 days 0:00:00.000
34:1950 0 days 0:00:00.000
31:1910 0 days 0:00:00.000
27:14b8 0 days 0:00:00.000
26:14b4 0 days 0:00:00.000
25:14ac 0 days 0:00:00.000
24:13d0 0 days 0:00:00.000
23:13c8 0 days 0:00:00.000
22:13b4 0 days 0:00:00.000
21:13b0 0 days 0:00:00.000
20:13a8 0 days 0:00:00.000
19:1328 0 days 0:00:00.000
16:e20 0 days 0:00:00.000
15:e10 0 days 0:00:00.000
14:dd0 0 days 0:00:00.000
13:ce0 0 days 0:00:00.000
12:754 0 days 0:00:00.000
11:718 0 days 0:00:00.000
10:678 0 days 0:00:00.000
9:674 0 days 0:00:00.000
8:668 0 days 0:00:00.000
5:548 0 days 0:00:00.000
2:4a0 0 days 0:00:00.000
1:418 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Kernel Mode Time
Thread Time
32:14a0 0 days 0:02:24.878
33:b6c 0 days 0:00:00.171
30:fa8 0 days 0:00:00.109
40:1bd8 0 days 0:00:00.093
43:11d0 0 days 0:00:00.078
42:4ac 0 days 0:00:00.062
34:1950 0 days 0:00:00.062
7:664 0 days 0:00:00.062
41:174c 0 days 0:00:00.046
28:ff8 0 days 0:00:00.031
18:1304 0 days 0:00:00.031
16:e20 0 days 0:00:00.031
1:418 0 days 0:00:00.031
45:650 0 days 0:00:00.015
17:f68 0 days 0:00:00.015
6:660 0 days 0:00:00.015
4:518 0 days 0:00:00.015
2:4a0 0 days 0:00:00.015
47:9c8 0 days 0:00:00.000
46:f20 0 days 0:00:00.000
44:1440 0 days 0:00:00.000
39:990 0 days 0:00:00.000
38:db4 0 days 0:00:00.000
37:6ac 0 days 0:00:00.000
36:c4c 0 days 0:00:00.000
35:ff4 0 days 0:00:00.000
31:1910 0 days 0:00:00.000
29:e0c 0 days 0:00:00.000
27:14b8 0 days 0:00:00.000
26:14b4 0 days 0:00:00.000
25:14ac 0 days 0:00:00.000
24:13d0 0 days 0:00:00.000
23:13c8 0 days 0:00:00.000
22:13b4 0 days 0:00:00.000
21:13b0 0 days 0:00:00.000
20:13a8 0 days 0:00:00.000
19:1328 0 days 0:00:00.000
15:e10 0 days 0:00:00.000
14:dd0 0 days 0:00:00.000
13:ce0 0 days 0:00:00.000
12:754 0 days 0:00:00.000
11:718 0 days 0:00:00.000
10:678 0 days 0:00:00.000
9:674 0 days 0:00:00.000
8:668 0 days 0:00:00.000
5:548 0 days 0:00:00.000
3:4a4 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Elapsed Time
Thread Time
0:410 0 days 0:35:17.550
1:418 0 days 0:35:17.534
3:4a4 0 days 0:35:17.331
2:4a0 0 days 0:35:17.331
4:518 0 days 0:35:16.957
5:548 0 days 0:35:16.817
6:660 0 days 0:35:15.176
7:664 0 days 0:35:15.156
8:668 0 days 0:35:15.126
9:674 0 days 0:35:15.026
10:678 0 days 0:35:14.986
11:718 0 days 0:35:13.066
12:754 0 days 0:35:13.056
13:ce0 0 days 0:35:08.131
14:dd0 0 days 0:35:07.641
15:e10 0 days 0:35:07.551
16:e20 0 days 0:35:07.531
17:f68 0 days 0:35:06.611
18:1304 0 days 0:35:03.553
19:1328 0 days 0:35:03.381
20:13a8 0 days 0:35:02.024
22:13b4 0 days 0:35:01.931
21:13b0 0 days 0:35:01.931
24:13d0 0 days 0:35:01.899
23:13c8 0 days 0:35:01.899
27:14b8 0 days 0:35:00.932
26:14b4 0 days 0:35:00.932
25:14ac 0 days 0:35:00.932
28:ff8 0 days 0:33:04.785
29:e0c 0 days 0:33:04.644
30:fa8 0 days 0:32:59.750
31:1910 0 days 0:32:55.608
32:14a0 0 days 0:32:12.276
34:1950 0 days 0:18:39.607
33:b6c 0 days 0:18:39.607
35:ff4 0 days 0:17:43.530
36:c4c 0 days 0:08:45.458
38:db4 0 days 0:07:41.551
37:6ac 0 days 0:07:41.551
39:990 0 days 0:06:54.877
40:1bd8 0 days 0:06:54.867
41:174c 0 days 0:05:38.282
42:4ac 0 days 0:03:54.627
43:11d0 0 days 0:03:53.122
44:1440 0 days 0:03:51.627
45:650 0 days 0:02:15.536
47:9c8 0 days 0:00:11.100
46:f20 0 days 0:00:11.100

0:000> ~32k
# Child-SP RetAddr Call Site
00 00000000`0414f558 000007fe`fd4f1430 ntdll!NtWaitForMultipleObjects+0xa
01 00000000`0414f560 00000000`775706e0 KERNELBASE!WaitForMultipleObjectsEx+0xe8
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll -
02 00000000`0414f660 000007fe`ee7de250 kernel32!WaitForMultipleObjects+0xb0
03 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0x4e80
04 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd
05 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Process heap doesn’t have any large allocated virtual block with only almost 100 KB committed normal blocks:

0:000> !heap -s

******************************
NT HEAP STATS BELOW
******************************
LFH Key : 0x000000a57ddeb5ed
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000000340000 00000002 210688 94444 210688 85646 752 86 0 1 LFH
External fragmentation 90 % (752 free blocks)
0000000000010000 00008000 64 4 64 1 1 1 0 0
00000000005c0000 00001002 3136 1256 3136 282 39 3 0 0 LFH
External fragmentation 22 % (39 free blocks)
0000000000960000 00001002 512 8 512 2 1 1 0 0
0000000001520000 00001002 512 280 512 3 10 1 0 0 LFH
00000000013f0000 00001002 512 8 512 2 1 1 0 0
0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH
00000000010b0000 00001002 512 8 512 2 1 1 0 0
00000000019b0000 00001002 512 320 512 6 12 1 0 0 LFH
0000000001220000 00001002 512 380 512 63 22 1 0 0 LFH
00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH
0000000001900000 00001002 512 156 512 133 1 1 0 0
00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH
0000000002e00000 00001002 48256 16656 48256 9870 101 13 0 1 LFH
External fragmentation 59 % (101 free blocks)
0000000002f10000 00001002 3584 3280 3584 1805 73 3 0 5 LFH
External fragmentation 55 % (73 free blocks)
00000000035c0000 00001002 64 8 64 3 1 1 0 0
0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH
00000000038c0000 00001002 512 8 512 3 1 1 0 0
0000000003ad0000 00001002 512 8 512 3 1 1 0 0
0000000003d40000 00001002 512 8 512 3 1 1 0 0
00000000005b0000 00001002 64 8 64 3 1 1 0 0
0000000004280000 00011002 512 64 512 59 2 1 0 0
00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH
0000000004270000 00001002 1088 304 1088 22 6 2 0 0 LFH
0000000007050000 00001002 512 8 512 3 1 1 0 0
0000000007cc0000 00001002 512 8 512 3 1 1 0 0
0000000000be0000 00001002 64 24 64 17 2 1 0 0
Virtual block: 0000000009b90000 - 0000000009b90000 (size 0000000000000000)
0000000006540000 00001002 1536 1064 1536 775 11 2 1 0 LFH
External fragmentation 72 % (11 free blocks)
0000000000b40000 00001002 64 8 64 3 1 1 0 0
0000000004950000 00001002 512 312 512 55 12 1 0 0 LFH
————————————————————————————-

So we consider the dump set as an instance of Memory Fluctuation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 117)

Thursday, October 22nd, 2015

Typical software trace may contain several Error Messages with different error codes and different exception names with Exception Stack Traces. Searching for individual codes or exceptions in problem databases may show many matches. Searching for all of them may show nothing. Therefore, we can construct the set of all subsets from the set of codes and exceptions (a power set) and perform analytic reasoning (and a search) based on certain subsets based on the problem description, Trace Viewpoints such as Use Case Trails, Motifs, Focus of Tracing, Foreground Components, (Adjoint) Threads of Activity, and simply some Activity Regions and Message Sets.

The following picture illustrates Error Powerset analysis pattern with a trace that has 4 error messages where 2 messages have the same error code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 116)

Sunday, October 18th, 2015

Sometimes we need to know about client-server interaction between components, threads, or processes in order to find out where the problem started. For example, if we have Error Message or Discontinuity in one PID Adjoint Thread of Activity, and we know that that process uses API from another PID, we can look at the latter PID Adjoint Thread to see if there are any Error Messages or other problems. The failure in the server can propagate to the client as illustrated in the following diagram:

We call this pattern Coupled Activities similar to Coupled Processes memory analysis pattern. It can help in Intra- and Inter-Correlation analysis, for example in choosing adjoint threads from Sheaf of Activities.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 230)

Wednesday, October 14th, 2015

Sometimes, a stack trace from Stack Trace Collection may look well-formed at the first sight like having an expected start frames, for example:

0:000> ~*k

[...]

# 19 Id: 16a4.21f4 Suspend: 0 Teb: 7e95b000 Unfrozen
ChildEBP RetAddr
0c2de6b0 74eb112f ntdll!NtWaitForMultipleObjects+0xc
0c2de83c 76ca7b89 KERNELBASE!WaitForMultipleObjectsEx+0xcc
0c2de858 76d007bf kernel32!WaitForMultipleObjects+0x19
0c2dec98 76d00295 kernel32!WerpReportFaultInternal+0x50b
0c2deca8 76ce1709 kernel32!WerpReportFault+0x74
0c2decb0 74f5f705 kernel32!BasepReportFault+0x19
0c2ded3c 76fb4f84 KERNELBASE!UnhandledExceptionFilter+0x1f4
0c2ded54 76fb5728 ntdll!TppExceptionFilter+0x30
0c2ded64 76f5c95a ntdll!TppWorkerpInnerExceptionFilter+0xe
0c2df914 76ca7c04 ntdll!TppWorkerThread+0×87f5a
0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0×24
0c2df970 76f1acea ntdll!__RtlUserThreadStart+0×2f
0c2df980 00000000 ntdll!_RtlUserThreadStart+0×1b

[...]

So, we may think something wrong happened in ntdll!TppWorkerThread code (although 0×87f5a offset looks suspicious). However, in reality, in this case due to exception filter logic (or some other reason in different cases) we have Hidden Stack Trace. When looking at UnhandledExceptionFilter parameters (or raw stack as in the case of Hidden Exceptions) we find an exception context:

0:019> kv
ChildEBP RetAddr Args to Child
0c2de6b0 74eb112f 00000003 0c2de880 00000001 ntdll!NtWaitForMultipleObjects+0xc
0c2de83c 76ca7b89 00000003 0c2de880 00000000 KERNELBASE!WaitForMultipleObjectsEx+0xcc
0c2de858 76d007bf 00000003 0c2de880 00000000 kernel32!WaitForMultipleObjects+0x19
0c2dec98 76d00295 00000000 00000001 00000000 kernel32!WerpReportFaultInternal+0x50b
0c2deca8 76ce1709 0c2ded3c 74f5f705 0c2ded94 kernel32!WerpReportFault+0x74
0c2decb0 74f5f705 0c2ded94 00000001 a79b7895 kernel32!BasepReportFault+0x19
0c2ded3c 76fb4f84 0c2ded94 0c2ded94 00000000 KERNELBASE!UnhandledExceptionFilter+0×1f4
0c2ded54 76fb5728 00000000 00000000 0c2df914 ntdll!TppExceptionFilter+0×30
0c2ded64 76f5c95a 0c2df8d0 76f00a70 0c2df914 ntdll!TppWorkerpInnerExceptionFilter+0xe
0c2df914 76ca7c04 0f79e380 76ca7be0 a5b45024 ntdll!TppWorkerThread+0×87f5a
0c2df928 76f1ad1f 0f79e380 a59141d7 00000000 kernel32!BaseThreadInitThunk+0×24
0c2df970 76f1acea ffffffff 76f00233 00000000 ntdll!__RtlUserThreadStart+0×2f
0c2df980 00000000 76ed4a00 0f79e380 00000000 ntdll!_RtlUserThreadStart+0×1b

0:019> dd 0c2ded94 L2
0c2ded94 0c2deef8 0c2def48

0:019> .cxr 0c2def48
eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10
eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
ntdll!RtlEnterCriticalSection+0×12:
76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

0:019> k
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0c2df3ac 7407999c ntdll!RtlEnterCriticalSection+0x12
0c2df3cc 7407acd6 ModuleA!DoWork+0x1b
[...]
0c2df73c 76ee3aa7 ModuleA!ThreadPoolWorkCallback+0xa9
0c2df77c 76ee1291 ntdll!TppWorkpExecuteCallback+0x137
0c2df914 76ca7c04 ntdll!TppWorkerThread+0x48e
0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0x24
0c2df970 76f1acea ntdll!__RtlUserThreadStart+0x2f
0c2df980 00000000 ntdll!_RtlUserThreadStart+0x1b

We consider this a different pattern than Hidden Call because an entire stack (sub)trace is missing between UnhandledExceptionFilter and thread start frames:

ntdll!NtWaitForMultipleObjects
KERNELBASE!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!BasepReportFault
KERNELBASE!UnhandledExceptionFilter
[...]
ntdll!TppWorkerThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

This pattern is also different from Past Stack Trace pattern because Hidden Stack Trace belongs to PRESENT time zone. Our example is also different from Hidden Exception analysis pattern and its recovered stack trace because exception processing is not hidden and shows Exception Stack Trace albeit with a hidden part.

We were also fortunate to have Stored Exception (accessible by !analyze -v command):

0:019> .exr -1
ExceptionAddress: 76f00fb2 (ntdll!RtlEnterCriticalSection+0x00000012)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 15f237e5
Attempt to write to address 15f237e5

0:019> .ecxr
eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10
eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
ntdll!RtlEnterCriticalSection+0x12:
76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

So this Hidden Stack Trace is detected straightforwardly. But in other cases, such as when we have Multiple Exceptions in a process dump or Stack Trace Collection from a complete memory dump, we need to pay attention to such a possibility.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 115)

Monday, October 12th, 2015

By analogy with Adjoint Thread of Activity we introduce Adjoint Message analysis pattern. Most if not all analysis patterns focus on log message text and consider TID, PID, Module, source file and function as its attributes. However, we can choose one of attributes and consider it as a message in its own right with the original message text consigned now as another attribute. Then we can analyze the structure of the trace from the perspective of that newly selected message:

Since the number of different message values now is smaller (for example, module names) compared to normal trace messages we can use them in protein-like encoding and structure analysis schemes (see Software Trace and Logs as Proteins). We metaphorically name Adjoint Messages as Amino-acid-Messages (A-Messages). We can also compress same message sequences into one message which may be useful for pattern matching (and even use different color intensities to represent message cardinalities):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -