Spiking Thread, Top Module, Module Hint, and Memory Fluctuation: pattern cooperation

We noticed that after restarting Windows 7 system on one of our notebooks it becomes very sluggish. Task Manager showed 25% CPU usage in one of svchost.exe processes and very high usage of physical memory. So we immediately dumped it using procdump. The resulted process memory dump was almost 1.5 GB. Although the analysis case is very simple and straightforward we decided to publish to show the value of crash and hang dump analysis in understanding abnormal software behavior in “user” context.

When we open the memory dump and run !runaway WinDbg command we immediately recognize Spiking Thread pattern:

0:000> !runaway f
User Mode Time
Thread Time
38:14a0 0 days 0:04:59.911
36:a88 0 days 0:00:00.187
2:41c 0 days 0:00:00.187
39:19f8 0 days 0:00:00.046
34:fa8 0 days 0:00:00.046
21:12b4 0 days 0:00:00.031
43:8f0 0 days 0:00:00.015
42:1504 0 days 0:00:00.015
41:1a20 0 days 0:00:00.015
40:978 0 days 0:00:00.015
33:e0c 0 days 0:00:00.015
32:ff8 0 days 0:00:00.015
22:1304 0 days 0:00:00.015
19:f68 0 days 0:00:00.015
9:664 0 days 0:00:00.015
8:660 0 days 0:00:00.015
6:518 0 days 0:00:00.015
4:4a4 0 days 0:00:00.015
51:160c 0 days 0:00:00.000
50:1590 0 days 0:00:00.000
49:15d8 0 days 0:00:00.000
48:ac8 0 days 0:00:00.000
47:14d0 0 days 0:00:00.000
46:1bfc 0 days 0:00:00.000
45:18e8 0 days 0:00:00.000
44:1448 0 days 0:00:00.000
37:1910 0 days 0:00:00.000
35:558 0 days 0:00:00.000
31:14b8 0 days 0:00:00.000
30:14b4 0 days 0:00:00.000
29:14ac 0 days 0:00:00.000
28:13d0 0 days 0:00:00.000
27:13c8 0 days 0:00:00.000
26:13b4 0 days 0:00:00.000
25:13b0 0 days 0:00:00.000
24:13a8 0 days 0:00:00.000
23:1328 0 days 0:00:00.000
20:12b0 0 days 0:00:00.000
18:e20 0 days 0:00:00.000
17:e10 0 days 0:00:00.000
16:dd0 0 days 0:00:00.000
15:ce0 0 days 0:00:00.000
14:754 0 days 0:00:00.000
13:718 0 days 0:00:00.000
12:678 0 days 0:00:00.000
11:674 0 days 0:00:00.000
10:668 0 days 0:00:00.000
7:548 0 days 0:00:00.000
5:4ac 0 days 0:00:00.000
3:4a0 0 days 0:00:00.000
1:418 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Kernel Mode Time
Thread Time
38:14a0 0 days 0:00:55.707
36:a88 0 days 0:00:01.778
2:41c 0 days 0:00:00.405
34:fa8 0 days 0:00:00.109
9:664 0 days 0:00:00.062
43:8f0 0 days 0:00:00.046
42:1504 0 days 0:00:00.046
21:12b4 0 days 0:00:00.046
32:ff8 0 days 0:00:00.031
22:1304 0 days 0:00:00.031
18:e20 0 days 0:00:00.031
39:19f8 0 days 0:00:00.015
19:f68 0 days 0:00:00.015
6:518 0 days 0:00:00.015
3:4a0 0 days 0:00:00.015
1:418 0 days 0:00:00.015
51:160c 0 days 0:00:00.000
50:1590 0 days 0:00:00.000
49:15d8 0 days 0:00:00.000
48:ac8 0 days 0:00:00.000
47:14d0 0 days 0:00:00.000
46:1bfc 0 days 0:00:00.000
45:18e8 0 days 0:00:00.000
44:1448 0 days 0:00:00.000
41:1a20 0 days 0:00:00.000
40:978 0 days 0:00:00.000
37:1910 0 days 0:00:00.000
35:558 0 days 0:00:00.000
33:e0c 0 days 0:00:00.000
31:14b8 0 days 0:00:00.000
30:14b4 0 days 0:00:00.000
29:14ac 0 days 0:00:00.000
28:13d0 0 days 0:00:00.000
27:13c8 0 days 0:00:00.000
26:13b4 0 days 0:00:00.000
25:13b0 0 days 0:00:00.000
24:13a8 0 days 0:00:00.000
23:1328 0 days 0:00:00.000
20:12b0 0 days 0:00:00.000
17:e10 0 days 0:00:00.000
16:dd0 0 days 0:00:00.000
15:ce0 0 days 0:00:00.000
14:754 0 days 0:00:00.000
13:718 0 days 0:00:00.000
12:678 0 days 0:00:00.000
11:674 0 days 0:00:00.000
10:668 0 days 0:00:00.000
8:660 0 days 0:00:00.000
7:548 0 days 0:00:00.000
5:4ac 0 days 0:00:00.000
4:4a4 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Elapsed Time
Thread Time
0:410 0 days 0:10:24.550
2:41c 0 days 0:10:24.534
1:418 0 days 0:10:24.534
4:4a4 0 days 0:10:24.331
3:4a0 0 days 0:10:24.331
5:4ac 0 days 0:10:24.269
6:518 0 days 0:10:23.957
7:548 0 days 0:10:23.817
8:660 0 days 0:10:22.176
9:664 0 days 0:10:22.156
10:668 0 days 0:10:22.126
11:674 0 days 0:10:22.026
12:678 0 days 0:10:21.986
13:718 0 days 0:10:20.066
14:754 0 days 0:10:20.056
15:ce0 0 days 0:10:15.131
16:dd0 0 days 0:10:14.641
17:e10 0 days 0:10:14.551
18:e20 0 days 0:10:14.531
19:f68 0 days 0:10:13.611
21:12b4 0 days 0:10:10.647
20:12b0 0 days 0:10:10.647
22:1304 0 days 0:10:10.553
23:1328 0 days 0:10:10.381
24:13a8 0 days 0:10:09.024
26:13b4 0 days 0:10:08.931
25:13b0 0 days 0:10:08.931
28:13d0 0 days 0:10:08.899
27:13c8 0 days 0:10:08.899
31:14b8 0 days 0:10:07.932
30:14b4 0 days 0:10:07.932
29:14ac 0 days 0:10:07.932
32:ff8 0 days 0:08:11.785
33:e0c 0 days 0:08:11.644
34:fa8 0 days 0:08:06.750
35:558 0 days 0:08:05.765
36:a88 0 days 0:08:05.127
37:1910 0 days 0:08:02.608
38:14a0 0 days 0:07:19.276
42:1504 0 days 0:04:55.634
41:1a20 0 days 0:04:55.634
40:978 0 days 0:04:55.634
39:19f8 0 days 0:04:55.634
43:8f0 0 days 0:04:55.618
44:1448 0 days 0:04:42.634
46:1bfc 0 days 0:04:20.945
45:18e8 0 days 0:04:20.945
47:14d0 0 days 0:02:42.515
48:ac8 0 days 0:01:27.434
50:1590 0 days 0:00:04.917
49:15d8 0 days 0:00:04.917
51:160c 0 days 0:00:03.387

We see that most of the time the thread was consuming CPU in user mode and less in kernel mode.

We then examine that thread stack trace:

0:000> ~38k
# Child-SP RetAddr Call Site
00 00000000`0414ddd8 00000000`7769cf66 ntdll!ZwQueryPerformanceCounter+0xa
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll -
01 00000000`0414dde0 000007fe`ee7ee7aa ntdll!RtlQueryPerformanceFrequency+0x16
02 00000000`0414de10 000007fe`ee7ee53f wuaueng!DllInstall+0×153da
03 00000000`0414de40 000007fe`ee7e0d99 wuaueng!DllInstall+0×1516f
04 00000000`0414e190 000007fe`ee7df542 wuaueng!DllInstall+0×79c9
05 00000000`0414e1f0 000007fe`ee7df57c wuaueng!DllInstall+0×6172
06 00000000`0414e2b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
07 00000000`0414e370 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
08 00000000`0414e430 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
09 00000000`0414e4f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0a 00000000`0414e5b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0b 00000000`0414e670 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0c 00000000`0414e730 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0d 00000000`0414e7f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0e 00000000`0414e8b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
0f 00000000`0414e970 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
10 00000000`0414ea30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
11 00000000`0414eaf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
12 00000000`0414ebb0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
13 00000000`0414ec70 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
14 00000000`0414ed30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
15 00000000`0414edf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac
16 00000000`0414eeb0 000007fe`ee7df2d7 wuaueng!DllInstall+0×61ac
17 00000000`0414ef70 000007fe`ee7e9b4f wuaueng!DllInstall+0×5f07
18 00000000`0414f020 000007fe`ee7eb7e8 wuaueng!DllInstall+0×1077f
19 00000000`0414f370 000007fe`ee8010b2 wuaueng!DllInstall+0×12418
1a 00000000`0414f5b0 000007fe`ee7fe53e wuaueng!DllInstall+0×27ce2
1b 00000000`0414f610 000007fe`ee7fccac wuaueng!DllInstall+0×2516e
1c 00000000`0414f660 000007fe`ee7dec19 wuaueng!DllInstall+0×238dc
1d 00000000`0414f690 000007fe`ee7de30f wuaueng!DllInstall+0×5849
1e 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0×4f3f
1f 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd
20 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that Top Module is from Windows Update Agent:

0:000> lmv m wuaueng
Browse full module list
start end module name
000007fe`ee740000 000007fe`ee9a0000 wuaueng (export symbols) wuaueng.dll
Loaded symbol image file: wuaueng.dll
Image path: c:\Windows\System32\wuaueng.dll
Image name: wuaueng.dll
Browse all global symbols functions data
Timestamp: Wed May 14 17:21:24 2014 (53739804)
CheckSum: 00265DEA
ImageSize: 00260000
File version: 7.6.7600.320
Product version: 7.6.7600.320
File flags: 8 (Mask 3F) Private
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: wuaueng.dll
OriginalFilename: wuaueng.dll
ProductVersion: 7.6.7600.320
FileVersion: 7.6.7600.320 (winmain_wtr_wsus3sp2(oobla).140514-0916)
FileDescription: Windows Update Agent
LegalCopyright: © Microsoft Corporation. All rights reserved.

We also examine process address space:

0:000> !address -summary

Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 507 7ff`9373b000 ( 7.998 TB) 99.98%
Heap 574 0`3e2ad000 ( 994.676 MB) 57.27% 0.01%
9786 0`2803e000 ( 640.242 MB) 36.87% 0.01%
Image 874 0`049a6000 ( 73.648 MB) 4.24% 0.00%
Stack 156 0`01a00000 ( 26.000 MB) 1.50% 0.00%
Other 13 0`001bb000 ( 1.730 MB) 0.10% 0.00%
TEB 52 0`00068000 ( 416.000 kB) 0.02% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 834 0`5345e000 ( 1.301 GB) 76.72% 0.02%
MEM_MAPPED 9747 0`14ab0000 ( 330.688 MB) 19.04% 0.00%
MEM_IMAGE 875 0`049a7000 ( 73.652 MB) 4.24% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 507 7ff`9373b000 ( 7.998 TB) 99.98%
MEM_COMMIT 11164 0`513d8000 ( 1.269 GB) 74.85% 0.02%
MEM_RESERVE 292 0`1b4dd000 ( 436.863 MB) 25.15% 0.01%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 5533 0`41d6c000 ( 1.029 GB) 60.66% 0.01%
PAGE_WRITECOPY 4911 0`098b9000 ( 152.723 MB) 8.79% 0.00%
PAGE_READONLY 504 0`03204000 ( 50.016 MB) 2.88% 0.00%
PAGE_EXECUTE_READ 164 0`02b29000 ( 43.160 MB) 2.49% 0.00%
PAGE_READWRITE|PAGE_GUARD 52 0`00086000 ( 536.000 kB) 0.03% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0`ff84b000 7fd`eed65000 ( 7.992 TB)
Heap 0`83eb0000 0`01c17000 ( 28.090 MB)
0`93070000 0`0ffbd000 ( 255.738 MB)
Image 7fe`fe18a000 0`0089e000 ( 8.617 MB)
Stack 0`07140000 0`0007b000 ( 492.000 kB)
Other 0`00760000 0`00181000 ( 1.504 MB)
TEB 7ff`fff0a000 0`00002000 ( 8.000 kB)
PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

We see that process heap occupies almost 1 GB. Let’s see its statistics:

0:000> !heap -s

******************************
NT HEAP STATS BELOW
******************************
LFH Key : 0x000000a57ddeb5ed
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
Virtual block: 0000000051eb0000 - 0000000051eb0000 (size 0000000000000000)
Virtual block: 0000000052930000 - 0000000052930000 (size 0000000000000000)
Virtual block: 0000000026ba0000 - 0000000026ba0000 (size 0000000000000000)
Virtual block: 0000000053e50000 - 0000000053e50000 (size 0000000000000000)
Virtual block: 0000000054490000 - 0000000054490000 (size 0000000000000000)
Virtual block: 00000000547a0000 - 00000000547a0000 (size 0000000000000000)
Virtual block: 000000003bda0000 - 000000003bda0000 (size 0000000000000000)
Virtual block: 0000000056030000 - 0000000056030000 (size 0000000000000000)
Virtual block: 00000000567c0000 - 00000000567c0000 (size 0000000000000000)
Virtual block: 00000000572a0000 - 00000000572a0000 (size 0000000000000000)
Virtual block: 0000000057870000 - 0000000057870000 (size 0000000000000000)
Virtual block: 0000000045a10000 - 0000000045a10000 (size 0000000000000000)
Virtual block: 0000000058fb0000 - 0000000058fb0000 (size 0000000000000000)
Virtual block: 0000000045c10000 - 0000000045c10000 (size 0000000000000000)
Virtual block: 00000000599c0000 - 00000000599c0000 (size 0000000000000000)
Virtual block: 0000000059ff0000 - 0000000059ff0000 (size 0000000000000000)
Virtual block: 000000005ae20000 - 000000005ae20000 (size 0000000000000000)
Virtual block: 000000005c5d0000 - 000000005c5d0000 (size 0000000000000000)
Virtual block: 0000000054b90000 - 0000000054b90000 (size 0000000000000000)
Virtual block: 000000005d070000 - 000000005d070000 (size 0000000000000000)
Virtual block: 000000006e370000 - 000000006e370000 (size 0000000000000000)
Virtual block: 000000006f8e0000 - 000000006f8e0000 (size 0000000000000000)
Virtual block: 000000006ed20000 - 000000006ed20000 (size 0000000000000000)
Virtual block: 0000000070890000 - 0000000070890000 (size 0000000000000000)
Virtual block: 000000005e370000 - 000000005e370000 (size 0000000000000000)
Virtual block: 000000005f5a0000 - 000000005f5a0000 (size 0000000000000000)
Virtual block: 000000005fa60000 - 000000005fa60000 (size 0000000000000000)
Virtual block: 000000005ffe0000 - 000000005ffe0000 (size 0000000000000000)
Virtual block: 0000000060770000 - 0000000060770000 (size 0000000000000000)
Virtual block: 0000000060aa0000 - 0000000060aa0000 (size 0000000000000000)
Virtual block: 0000000061810000 - 0000000061810000 (size 0000000000000000)
Virtual block: 0000000061a30000 - 0000000061a30000 (size 0000000000000000)
Virtual block: 0000000061f00000 - 0000000061f00000 (size 0000000000000000)
Virtual block: 0000000064470000 - 0000000064470000 (size 0000000000000000)
Virtual block: 0000000064c00000 - 0000000064c00000 (size 0000000000000000)
Virtual block: 00000000656d0000 - 00000000656d0000 (size 0000000000000000)
Virtual block: 00000000660f0000 - 00000000660f0000 (size 0000000000000000)
Virtual block: 0000000066530000 - 0000000066530000 (size 0000000000000000)
Virtual block: 00000000669d0000 - 00000000669d0000 (size 0000000000000000)
Virtual block: 00000000676a0000 - 00000000676a0000 (size 0000000000000000)
Virtual block: 0000000067a70000 - 0000000067a70000 (size 0000000000000000)
Virtual block: 0000000068a20000 - 0000000068a20000 (size 0000000000000000)
Virtual block: 0000000069f10000 - 0000000069f10000 (size 0000000000000000)
Virtual block: 000000006a6c0000 - 000000006a6c0000 (size 0000000000000000)
Virtual block: 000000006ad80000 - 000000006ad80000 (size 0000000000000000)
Virtual block: 000000006b9a0000 - 000000006b9a0000 (size 0000000000000000)
Virtual block: 000000006bb40000 - 000000006bb40000 (size 0000000000000000)
Virtual block: 000000006c4f0000 - 000000006c4f0000 (size 0000000000000000)
Virtual block: 000000006dc30000 - 000000006dc30000 (size 0000000000000000)
Virtual block: 000000006de10000 - 000000006de10000 (size 0000000000000000)
Virtual block: 000000006ef80000 - 000000006ef80000 (size 0000000000000000)
Virtual block: 00000000728f0000 - 00000000728f0000 (size 0000000000000000)
Virtual block: 0000000071270000 - 0000000071270000 (size 0000000000000000)
Virtual block: 0000000074030000 - 0000000074030000 (size 0000000000000000)
Virtual block: 00000000746c0000 - 00000000746c0000 (size 0000000000000000)
Virtual block: 00000000749d0000 - 00000000749d0000 (size 0000000000000000)
Virtual block: 0000000074e50000 - 0000000074e50000 (size 0000000000000000)
Virtual block: 0000000071730000 - 0000000071730000 (size 0000000000000000)
Virtual block: 0000000075c30000 - 0000000075c30000 (size 0000000000000000)
Virtual block: 0000000075e00000 - 0000000075e00000 (size 0000000000000000)
Virtual block: 0000000075fe0000 - 0000000075fe0000 (size 0000000000000000)
Virtual block: 00000000761a0000 - 00000000761a0000 (size 0000000000000000)
Virtual block: 0000000076460000 - 0000000076460000 (size 0000000000000000)
Virtual block: 0000000076620000 - 0000000076620000 (size 0000000000000000)
Virtual block: 0000000077850000 - 0000000077850000 (size 0000000000000000)
Virtual block: 0000000076c60000 - 0000000076c60000 (size 0000000000000000)
Virtual block: 0000000078ec0000 - 0000000078ec0000 (size 0000000000000000)
Virtual block: 0000000062b30000 - 0000000062b30000 (size 0000000000000000)
Virtual block: 000000007a520000 - 000000007a520000 (size 0000000000000000)
Virtual block: 000000007ab50000 - 000000007ab50000 (size 0000000000000000)
Virtual block: 00000000770e0000 - 00000000770e0000 (size 0000000000000000)
Virtual block: 000000007af00000 - 000000007af00000 (size 0000000000000000)
Virtual block: 000000007c6b0000 - 000000007c6b0000 (size 0000000000000000)
Virtual block: 000000007de50000 - 000000007de50000 (size 0000000000000000)
Virtual block: 000000007e2d0000 - 000000007e2d0000 (size 0000000000000000)
Virtual block: 000000007fff0000 - 000000007fff0000 (size 0000000000000000)
Virtual block: 00000000628c0000 - 00000000628c0000 (size 0000000000000000)
Virtual block: 00000000809b0000 - 00000000809b0000 (size 0000000000000000)
Virtual block: 0000000080d80000 - 0000000080d80000 (size 0000000000000000)
Virtual block: 0000000081310000 - 0000000081310000 (size 0000000000000000)
Virtual block: 00000000631a0000 - 00000000631a0000 (size 0000000000000000)
Virtual block: 0000000062c70000 - 0000000062c70000 (size 0000000000000000)
Virtual block: 0000000092ca0000 - 0000000092ca0000 (size 0000000000000000)
Virtual block: 0000000003940000 - 0000000003940000 (size 0000000000000000)
Virtual block: 0000000081cd0000 - 0000000081cd0000 (size 0000000000000000)
Virtual block: 0000000082260000 - 0000000082260000 (size 0000000000000000)
Virtual block: 0000000077200000 - 0000000077200000 (size 0000000000000000)
Virtual block: 000000007ec90000 - 000000007ec90000 (size 0000000000000000)
Virtual block: 000000007edb0000 - 000000007edb0000 (size 0000000000000000)
Virtual block: 000000007eec0000 - 000000007eec0000 (size 0000000000000000)
Virtual block: 00000000829f0000 - 00000000829f0000 (size 0000000000000000)
Virtual block: 0000000003770000 - 0000000003770000 (size 0000000000000000)
Virtual block: 0000000082fc0000 - 0000000082fc0000 (size 0000000000000000)
Virtual block: 0000000083210000 - 0000000083210000 (size 0000000000000000)
Virtual block: 0000000083320000 - 0000000083320000 (size 0000000000000000)
Virtual block: 0000000083760000 - 0000000083760000 (size 0000000000000000)
Virtual block: 0000000083870000 - 0000000083870000 (size 0000000000000000)
Virtual block: 0000000083eb0000 - 0000000083eb0000 (size 0000000000000000)
Virtual block: 0000000085ad0000 - 0000000085ad0000 (size 0000000000000000)
Virtual block: 0000000086290000 - 0000000086290000 (size 0000000000000000)

0000000000340000 00000002 210688 174796 210688 60711 775 45 100 1 LFH
External fragmentation 34 % (775 free blocks)
0000000000010000 00008000 64 4 64 1 1 1 0 0
00000000005c0000 00001002 3136 1256 3136 278 39 3 0 0 LFH
External fragmentation 22 % (39 free blocks)
0000000000960000 00001002 512 8 512 2 1 1 0 0
0000000001520000 00001002 512 276 512 2 10 1 0 0 LFH
00000000013f0000 00001002 512 8 512 2 1 1 0 0
0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH
00000000010b0000 00001002 512 8 512 2 1 1 0 0
00000000019b0000 00001002 512 308 512 5 8 1 0 0 LFH
0000000001220000 00001002 512 380 512 63 21 1 0 0 LFH
00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH
0000000001900000 00001002 512 156 512 133 1 1 0 0
00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH
0000000002e00000 00001002 48256 16656 48256 9836 98 13 0 1 LFH
External fragmentation 59 % (98 free blocks)
0000000002f10000 00001002 3584 3280 3584 1773 79 3 0 5 LFH
External fragmentation 54 % (79 free blocks)
00000000035c0000 00001002 64 8 64 3 1 1 0 0
0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH
00000000038c0000 00001002 512 8 512 3 1 1 0 0
0000000003ad0000 00001002 512 8 512 3 1 1 0 0
0000000003d40000 00001002 512 8 512 3 1 1 0 0
00000000005b0000 00001002 64 8 64 3 1 1 0 0
0000000004280000 00011002 512 64 512 59 2 1 0 0
00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH
0000000004270000 00001002 1088 304 1088 21 5 2 0 0 LFH
0000000006650000 00000002 1088 80 1088 10 2 2 0 0
0000000007050000 00001002 512 8 512 3 1 1 0 0
0000000007cc0000 00001002 512 8 512 3 1 1 0 0
0000000000be0000 00001002 64 24 64 17 2 1 0 0
0000000006540000 00001002 1536 544 1536 259 4 2 0 0 LFH
0000000000b40000 00001002 64 8 64 3 1 1 0 0
0000000004950000 00001002 512 8 512 3 1 1 0 0
————————————————————————————-

We see many large virtual blocks allocated with less than 200 KB of normal blocks committed. We suspect possible Memory Leak. Let’s check a few such virtual blocks:

0:000> !address 0000000062b30000

Usage: Heap
Base Address: 00000000`62b30000
End Address: 00000000`62c65000
Region Size: 00000000`00135000 ( 1.207 MB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000000`62b30000
Allocation Protect: 00000004 PAGE_READWRITE
More info: heap owning the address: !heap 0×340000
More info: heap large/virtual block
More info: heap entry containing the address: !heap -x 0×62b30000

Content source: 1 (target), length: 135000

0:000> !address 0000000003770000

Usage: Heap
Base Address: 00000000`03770000
End Address: 00000000`03870000
Region Size: 00000000`00100000 ( 1.000 MB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000000`03770000
Allocation Protect: 00000004 PAGE_READWRITE
More info: heap owning the address: !heap 0×340000
More info: heap large/virtual block
More info: heap entry containing the address: !heap -x 0×3770000

Content source: 1 (target), length: 100000

0:000> !address 0000000085ad0000

Usage: Heap
Base Address: 00000000`85ad0000
End Address: 00000000`8628d000
Region Size: 00000000`007bd000 ( 7.738 MB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000000`85ad0000
Allocation Protect: 00000004 PAGE_READWRITE
More info: heap owning the address: !heap 0×340000
More info: heap large/virtual block
More info: heap entry containing the address: !heap -x 0×85ad0000

Content source: 1 (target), length: 7bd000

We see they all contain assembly packages for KB articles for various updates (Module Hints):

0:000> dc 00000000`85ad0000
00000000`85ad0000 86290000 00000000 83eb0000 00000000 ..)………….
00000000`85ad0010 00000000 00000000 00000000 00000000 …………….
00000000`85ad0020 007bd000 00000000 007bd000 00000000 ..{…….{…..
00000000`85ad0030 00000000 00000000 f333297f 04000000 ………)3…..
00000000`85ad0040 00000001 007bc51a 00000004 00000000 ……{………
00000000`85ad0050 0061003c 00730073 006d0065 006c0062 <.a.s.s.e.m.b.l.
00000000`85ad0060 00200079 006d0078 006e006c 003d0073 y. .x.m.l.n.s.=.
00000000`85ad0070 00750022 006e0072 0073003a 00680063 “.u.r.n.:.s.c.h.

0:000> du 00000000`85ad0050
00000000`85ad0050 “<assembly xmlns=”urn:schemas-mic”
00000000`85ad0090 “rosoft-com:asm.v3″ manifestVersi”
00000000`85ad00d0 “on=”1.0″ description=”Fix for KB”
00000000`85ad0110 “2798162″ displayName=”default” c”
00000000`85ad0150 “ompany=”Microsoft Corporation” c”
00000000`85ad0190 “opyright=”Microsoft Corporation”"
00000000`85ad01d0 ” supportInformation=”http://supp”
00000000`85ad0210 “ort.microsoft.com/?kbid=2798162″”
00000000`85ad0250 ” creationTimeStamp=”2013-04-15T0″
00000000`85ad0290 “4:10:39Z” lastUpdateTimeStamp=”2″
00000000`85ad02d0 “013-04-15T04:10:39Z”><assemblyid”
00000000`85ad0310 “entity name=”Package_for_KB27981″

However, in less than 20 minutes CPU and memory consumption normalized and the new saved process memory dump was less than 150 KB. We open it and see that CPU consumption happened for about 15 minutes (in the past) with the consuming thread now #32 instead of #38:

0:000> !runaway f
User Mode Time
Thread Time
32:14a0 0 days 0:13:24.029
33:b6c 0 days 0:00:00.078
41:174c 0 days 0:00:00.062
40:1bd8 0 days 0:00:00.062
30:fa8 0 days 0:00:00.046
45:650 0 days 0:00:00.015
42:4ac 0 days 0:00:00.015
39:990 0 days 0:00:00.015
29:e0c 0 days 0:00:00.015
28:ff8 0 days 0:00:00.015
18:1304 0 days 0:00:00.015
17:f68 0 days 0:00:00.015
7:664 0 days 0:00:00.015
6:660 0 days 0:00:00.015
4:518 0 days 0:00:00.015
3:4a4 0 days 0:00:00.015
47:9c8 0 days 0:00:00.000
46:f20 0 days 0:00:00.000
44:1440 0 days 0:00:00.000
43:11d0 0 days 0:00:00.000
38:db4 0 days 0:00:00.000
37:6ac 0 days 0:00:00.000
36:c4c 0 days 0:00:00.000
35:ff4 0 days 0:00:00.000
34:1950 0 days 0:00:00.000
31:1910 0 days 0:00:00.000
27:14b8 0 days 0:00:00.000
26:14b4 0 days 0:00:00.000
25:14ac 0 days 0:00:00.000
24:13d0 0 days 0:00:00.000
23:13c8 0 days 0:00:00.000
22:13b4 0 days 0:00:00.000
21:13b0 0 days 0:00:00.000
20:13a8 0 days 0:00:00.000
19:1328 0 days 0:00:00.000
16:e20 0 days 0:00:00.000
15:e10 0 days 0:00:00.000
14:dd0 0 days 0:00:00.000
13:ce0 0 days 0:00:00.000
12:754 0 days 0:00:00.000
11:718 0 days 0:00:00.000
10:678 0 days 0:00:00.000
9:674 0 days 0:00:00.000
8:668 0 days 0:00:00.000
5:548 0 days 0:00:00.000
2:4a0 0 days 0:00:00.000
1:418 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Kernel Mode Time
Thread Time
32:14a0 0 days 0:02:24.878
33:b6c 0 days 0:00:00.171
30:fa8 0 days 0:00:00.109
40:1bd8 0 days 0:00:00.093
43:11d0 0 days 0:00:00.078
42:4ac 0 days 0:00:00.062
34:1950 0 days 0:00:00.062
7:664 0 days 0:00:00.062
41:174c 0 days 0:00:00.046
28:ff8 0 days 0:00:00.031
18:1304 0 days 0:00:00.031
16:e20 0 days 0:00:00.031
1:418 0 days 0:00:00.031
45:650 0 days 0:00:00.015
17:f68 0 days 0:00:00.015
6:660 0 days 0:00:00.015
4:518 0 days 0:00:00.015
2:4a0 0 days 0:00:00.015
47:9c8 0 days 0:00:00.000
46:f20 0 days 0:00:00.000
44:1440 0 days 0:00:00.000
39:990 0 days 0:00:00.000
38:db4 0 days 0:00:00.000
37:6ac 0 days 0:00:00.000
36:c4c 0 days 0:00:00.000
35:ff4 0 days 0:00:00.000
31:1910 0 days 0:00:00.000
29:e0c 0 days 0:00:00.000
27:14b8 0 days 0:00:00.000
26:14b4 0 days 0:00:00.000
25:14ac 0 days 0:00:00.000
24:13d0 0 days 0:00:00.000
23:13c8 0 days 0:00:00.000
22:13b4 0 days 0:00:00.000
21:13b0 0 days 0:00:00.000
20:13a8 0 days 0:00:00.000
19:1328 0 days 0:00:00.000
15:e10 0 days 0:00:00.000
14:dd0 0 days 0:00:00.000
13:ce0 0 days 0:00:00.000
12:754 0 days 0:00:00.000
11:718 0 days 0:00:00.000
10:678 0 days 0:00:00.000
9:674 0 days 0:00:00.000
8:668 0 days 0:00:00.000
5:548 0 days 0:00:00.000
3:4a4 0 days 0:00:00.000
0:410 0 days 0:00:00.000
Elapsed Time
Thread Time
0:410 0 days 0:35:17.550
1:418 0 days 0:35:17.534
3:4a4 0 days 0:35:17.331
2:4a0 0 days 0:35:17.331
4:518 0 days 0:35:16.957
5:548 0 days 0:35:16.817
6:660 0 days 0:35:15.176
7:664 0 days 0:35:15.156
8:668 0 days 0:35:15.126
9:674 0 days 0:35:15.026
10:678 0 days 0:35:14.986
11:718 0 days 0:35:13.066
12:754 0 days 0:35:13.056
13:ce0 0 days 0:35:08.131
14:dd0 0 days 0:35:07.641
15:e10 0 days 0:35:07.551
16:e20 0 days 0:35:07.531
17:f68 0 days 0:35:06.611
18:1304 0 days 0:35:03.553
19:1328 0 days 0:35:03.381
20:13a8 0 days 0:35:02.024
22:13b4 0 days 0:35:01.931
21:13b0 0 days 0:35:01.931
24:13d0 0 days 0:35:01.899
23:13c8 0 days 0:35:01.899
27:14b8 0 days 0:35:00.932
26:14b4 0 days 0:35:00.932
25:14ac 0 days 0:35:00.932
28:ff8 0 days 0:33:04.785
29:e0c 0 days 0:33:04.644
30:fa8 0 days 0:32:59.750
31:1910 0 days 0:32:55.608
32:14a0 0 days 0:32:12.276
34:1950 0 days 0:18:39.607
33:b6c 0 days 0:18:39.607
35:ff4 0 days 0:17:43.530
36:c4c 0 days 0:08:45.458
38:db4 0 days 0:07:41.551
37:6ac 0 days 0:07:41.551
39:990 0 days 0:06:54.877
40:1bd8 0 days 0:06:54.867
41:174c 0 days 0:05:38.282
42:4ac 0 days 0:03:54.627
43:11d0 0 days 0:03:53.122
44:1440 0 days 0:03:51.627
45:650 0 days 0:02:15.536
47:9c8 0 days 0:00:11.100
46:f20 0 days 0:00:11.100

0:000> ~32k
# Child-SP RetAddr Call Site
00 00000000`0414f558 000007fe`fd4f1430 ntdll!NtWaitForMultipleObjects+0xa
01 00000000`0414f560 00000000`775706e0 KERNELBASE!WaitForMultipleObjectsEx+0xe8
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll -
02 00000000`0414f660 000007fe`ee7de250 kernel32!WaitForMultipleObjects+0xb0
03 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0x4e80
04 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd
05 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Process heap doesn’t have any large allocated virtual block with only almost 100 KB committed normal blocks:

0:000> !heap -s

******************************
NT HEAP STATS BELOW
******************************
LFH Key : 0x000000a57ddeb5ed
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000000340000 00000002 210688 94444 210688 85646 752 86 0 1 LFH
External fragmentation 90 % (752 free blocks)
0000000000010000 00008000 64 4 64 1 1 1 0 0
00000000005c0000 00001002 3136 1256 3136 282 39 3 0 0 LFH
External fragmentation 22 % (39 free blocks)
0000000000960000 00001002 512 8 512 2 1 1 0 0
0000000001520000 00001002 512 280 512 3 10 1 0 0 LFH
00000000013f0000 00001002 512 8 512 2 1 1 0 0
0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH
00000000010b0000 00001002 512 8 512 2 1 1 0 0
00000000019b0000 00001002 512 320 512 6 12 1 0 0 LFH
0000000001220000 00001002 512 380 512 63 22 1 0 0 LFH
00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH
0000000001900000 00001002 512 156 512 133 1 1 0 0
00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH
0000000002e00000 00001002 48256 16656 48256 9870 101 13 0 1 LFH
External fragmentation 59 % (101 free blocks)
0000000002f10000 00001002 3584 3280 3584 1805 73 3 0 5 LFH
External fragmentation 55 % (73 free blocks)
00000000035c0000 00001002 64 8 64 3 1 1 0 0
0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH
00000000038c0000 00001002 512 8 512 3 1 1 0 0
0000000003ad0000 00001002 512 8 512 3 1 1 0 0
0000000003d40000 00001002 512 8 512 3 1 1 0 0
00000000005b0000 00001002 64 8 64 3 1 1 0 0
0000000004280000 00011002 512 64 512 59 2 1 0 0
00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH
0000000004270000 00001002 1088 304 1088 22 6 2 0 0 LFH
0000000007050000 00001002 512 8 512 3 1 1 0 0
0000000007cc0000 00001002 512 8 512 3 1 1 0 0
0000000000be0000 00001002 64 24 64 17 2 1 0 0
Virtual block: 0000000009b90000 - 0000000009b90000 (size 0000000000000000)
0000000006540000 00001002 1536 1064 1536 775 11 2 1 0 LFH
External fragmentation 72 % (11 free blocks)
0000000000b40000 00001002 64 8 64 3 1 1 0 0
0000000004950000 00001002 512 312 512 55 12 1 0 0 LFH
————————————————————————————-

So we consider the dump set as an instance of Memory Fluctuation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply

You must be logged in to post a comment.