Crash Dump Analysis Patterns (Part 20a)
Memory Leak is another pattern that may be finally manifested as Insufficient Memory pattern in a crash dump. In this part I’ll cover process heap memory leaks. They are usually identified when the process virtual memory size grows over time. It starts with 80Mb and instead of fluctuating normally below 100Mb it suddenly starts growing to 150Mb after some time and then to 300Mb next day and then grows to 600Mb and so on.
Usually a process heap is under suspicion here. To confirm this we need to sample 2-3 consecutive user memory dumps at process sizes 100Mb, 200Mb and 300Mb, for example. This can be done by using Microsoft userdump.exe command line tool. Then we can see whether there is any heap growth by using !heap -s WinDbg command:
1st dump
0:000> !heap -s
Heap Flags Reserv Commit Virt
(k) (k) (k)
---------------------------------------
00140000 00000002 2048 1048 1112
00240000 00008000 64 12 12
00310000 00001002 7232 4308 4600
00420000 00001002 1024 520 520
00340000 00001002 256 40 40
00720000 00001002 64 32 32
00760000 00001002 64 48 48
01020000 00001002 256 24 24
02060000 00001002 64 16 16
02070000 00001003 256 120 120
020b0000 00001003 256 4 4
020f0000 00001003 256 4 4
02130000 00001003 256 4 4
02170000 00001003 256 4 4
021f0000 00001002 1088 76 76
021e0000 00001002 64 16 16
02330000 00001002 1088 428 428
02340000 00011002 256 12 12
02380000 00001002 64 12 12
024c0000 00001003 64 8 8
028d0000 00001002 7232 3756 6188
02ce0000 00001003 64 8 8
07710000 00001002 64 20 20
07b20000 00001002 64 16 16
07f30000 00001002 64 16 16
09050000 00001002 256 12 12
09c80000 00001002 130304 102340 102684
007d0000 00001003 256 192 192
00810000 00001003 256 4 4
0bdd0000 00001003 256 4 4
0be10000 00001003 256 4 4
0be50000 00001003 256 4 4
0be90000 00001003 256 56 56
0bed0000 00001003 256 4 4
0bf10000 00001003 256 4 4
0bf50000 00001003 256 4 4
0bf90000 00001003 256 4 4
00860000 00001002 64 20 20
00870000 00001002 64 20 20
0d760000 00001002 256 12 12
0dc60000 00001002 1088 220 220
0c3a0000 00001002 64 12 12
0c3d0000 00001002 1088 160 364
08420000 00001002 64 64 64
2nd dump
0:000> !heap -s
Heap Flags Reserv Commit Virt
(k) (k) (k)
---------------------------------------
00140000 00000002 8192 4600 4600
00240000 00008000 64 12 12
00310000 00001002 7232 4516 4600
00420000 00001002 1024 520 520
00340000 00001002 256 44 44
00720000 00001002 64 32 32
00760000 00001002 64 48 48
01020000 00001002 256 24 24
02060000 00001002 64 16 16
02070000 00001003 256 124 124
020b0000 00001003 256 4 4
020f0000 00001003 256 4 4
02130000 00001003 256 4 4
02170000 00001003 256 4 4
021f0000 00001002 1088 76 76
021e0000 00001002 64 16 16
02330000 00001002 1088 428 428
02340000 00011002 256 12 12
02380000 00001002 64 12 12
024c0000 00001003 64 8 8
028d0000 00001002 7232 3796 6768
02ce0000 00001003 64 8 8
07710000 00001002 64 20 20
07b20000 00001002 64 16 16
07f30000 00001002 64 16 16
09050000 00001002 256 12 12
09c80000 00001002 261376 221152 221928
007d0000 00001003 256 192 192
00810000 00001003 256 4 4
0bdd0000 00001003 256 4 4
0be10000 00001003 256 4 4
0be50000 00001003 256 4 4
0be90000 00001003 256 60 60
0bed0000 00001003 256 4 4
0bf10000 00001003 256 4 4
0bf50000 00001003 256 4 4
0bf90000 00001003 256 4 4
00860000 00001002 64 20 20
00870000 00001002 64 20 20
0d760000 00001002 256 12 12
0dc60000 00001002 1088 228 228
0c3a0000 00001002 64 12 12
0c3d0000 00001002 1088 168 224
08450000 00001002 64 64 64
We see that the only significant heap growth is at 09c80000 address, from 130Mb to 260Mb. However this doesn’t say which code uses it. In order to find the code we need to enable the so called user mode stack trace database. Please refer to the following Citrix article:
http://support.citrix.com/article/CTX106970
The example in the article is for Citrix IMA service but you can replace ImaSrv.exe with any other executable name.
Suppose that after enabling user mode stack trace database and restarting the program or service we see the growth and we get memory dumps with the following suspicious heap highlighted in red:
0:000> !gflag
Current NtGlobalFlag contents: 0x00001000
ust - Create user mode stack trace database
0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key: 0x2687ed29
Heap Flags Reserv Commit Virt
(k) (k) (k)
---------------------------------------
00140000 58000062 4096 488 676
00240000 58008060 64 12 12
00360000 58001062 3136 1152 1216
003b0000 58001062 64 32 32
01690000 58001062 256 32 32
016d0000 58001062 1024 520 520
003e0000 58001062 64 48 48
02310000 58001062 256 24 24
02b30000 58001062 64 16 16
02b40000 58001063 256 64 64
02b80000 58001063 256 4 4
02bc0000 58001063 256 4 4
02c00000 58001063 256 4 4
02c40000 58001063 256 4 4
02c80000 58001063 256 4 4
02cc0000 58001063 256 4 4
02d30000 58001063 64 4 4
03140000 58001062 7232 4160 4896
03550000 58001063 64 4 4
07f70000 58001062 64 12 12
08380000 58001062 64 12 12
08790000 58001062 64 12 12
091d0000 58011062 256 12 12
09210000 58001062 64 16 16
09220000 58001062 64 12 12
092a0000 58001062 64 12 12
09740000 58001062 256 12 12
0b1a0000 58001062 64 12 12
0b670000 58001062 64768 39508 39700
0b7b0000 58001062 64 12 12
0c650000 58001062 1088 192 192
Every heap is subdivided into several segments and to see which segments have grown the most we can use !heap -m <heap address> command:
0:000> !heap -m 0b670000
Index Address Name Debugging options enabled
29: 0b670000
Segment at 0b670000 to 0b6b0000 (00040000 bytes committed)
Segment at 0c760000 to 0c860000 (00100000 bytes committed)
Segment at 0c980000 to 0cb80000 (001fe000 bytes committed)
Segment at 0cb80000 to 0cf80000 (003cc000 bytes committed)
Segment at 0dc30000 to 0e430000 (00800000 bytes committed)
Segment at 12330000 to 13330000 (01000000 bytes committed)
Segment at 13330000 to 15330000 (0078b000 bytes committed)
…
…
…
If we use !heap -a <heap address> command then in addition to the list of heap segments individual heap allocation entries will be dumped as well. This could be very big output and we should open the log file in advance by using .logopen <file name> command.
The output can be like this (taken from another dump):
0:000> !heap -a 000a0000
...
...
...
Segment00 at 000a0000:
Flags: 00000000
Base: 000a0000
First Entry: 000a0580
Last Entry: 000b0000
Total Pages: 00000010
Total UnCommit: 00000002
Largest UnCommit:00000000
UnCommitted Ranges: (1)
Heap entries for Segment00 in Heap 000a0000
000a0000: 00000 . 00580 [101] - busy (57f)
000a0580: 00580 . 00240 [101] - busy (23f)
000a07c0: 00240 . 00248 [101] - busy (22c)
000a0a08: 00248 . 00218 [101] - busy (200)
000a0c20: 00218 . 00ce0 [100]
000a1900: 00ce0 . 00f88 [101] - busy (f6a)
000a2888: 00f88 . 04418 [101] - busy (4400)
000a6ca0: 04418 . 05958 [101] - busy (5940)
000ac5f8: 05958 . 00928 [101] - busy (90c)
000acf20: 00928 . 010c0 [100]
000adfe0: 010c0 . 00020 [111] - busy (1d)
000ae000: 00002000 - uncommitted bytes.
Then we can inspect individual entries to see stack traces that allocated them by using !heap -p -a <heap entry address> command:
0:000> !heap -p -a 000a6ca0
address 000a6ca0 found in
_HEAP @ a0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
000a6ca0 0b2b 0000 [00] 000a6cb8 05940 - (busy)
Trace: 2156ac
7704dab4 ntdll!RtlAllocateHeap+0x0000021d
75c59b12 USP10!UspAllocCache+0x0000002b
75c62381 USP10!AllocSizeCache+0x00000048
75c61c74 USP10!FindOrCreateSizeCacheWithoutRealizationID+0x00000124
75c61bc0 USP10!FindOrCreateSizeCacheUsingRealizationID+0x00000070
75c59a97 USP10!UpdateCache+0x0000002b
75c59a61 USP10!ScriptCheckCache+0x0000005c
75c59d04 USP10!ScriptStringAnalyse+0x0000012a
7711140f LPK!LpkStringAnalyse+0x00000114
7711159e LPK!LpkCharsetDraw+0x00000302
77111488 LPK!LpkDrawTextEx+0x00000044
76a4beb3 USER32!DT_DrawStr+0x0000013a
76a4be45 USER32!DT_DrawJustifiedLine+0x0000005f
76a49d68 USER32!AddEllipsisAndDrawLine+0x00000186
76a4bc31 USER32!DrawTextExWorker+0x000001b1
76a4bedc USER32!DrawTextExW+0x0000001e
746051d8 uxtheme!CTextDraw::GetTextExtent+0x000000be
7460515a uxtheme!GetThemeTextExtent+0x00000065
74611ed4 uxtheme!CThemeMenuBar::MeasureItem+0x00000124
746119c1 uxtheme!CThemeMenu::OnMeasureItem+0x0000003f
74611978 uxtheme!CThemeWnd::_PreDefWindowProc+0x00000117
74601ea5 uxtheme!_ThemeDefWindowProc+0x00000090
74601f61 uxtheme!ThemeDefWindowProcW+0x00000018
76a4a09e USER32!DefWindowProcW+0x00000068
931406 notepad!NPWndProc+0x00000084
76a51a10 USER32!InternalCallWinProc+0x00000023
76a51ae8 USER32!UserCallWinProcCheckWow+0x0000014b
76a51c03 USER32!DispatchClientMessage+0x000000da
76a3bc24 USER32!__fnINOUTLPUAHMEASUREMENUITEM+0x00000027
77040e6e ntdll!KiUserCallbackDispatcher+0x0000002e
76a51d87 USER32!RealDefWindowProcW+0x00000047
74601f2f uxtheme!_ThemeDefWindowProc+0x000001b8
If we want to dump all heap entries with their corresponding stack traces we can use !heap -k -h <heap address> command.
Note: sometimes all these commands don’t work. In such cases we can use old Windows 2000 extension.
Some prefer to use umdh.exe and get text file logs but the advantage of embedding heap allocation stack traces in a dump is that we are not concerned with sending and configuring symbol files at a customer side.
- Dmitry Vostokov @ DumpAnalysis.org -
August 7th, 2007 at 8:59 am
You can also use Microsoft Debug Diagnostics Tool.
Sometimes it manages to pinpoint the leaking DLL, even without symbols.
see http://blogs.msdn.com/debugdiag/
August 19th, 2007 at 8:27 am
[…] the process size constantly grows but there is no difference in the process heap size. In such cases we need to check whether the process uses Microsoft .NET runtime (CLR). If one of […]
February 19th, 2008 at 10:55 am
various pageheap options !heap -p are useful such as (taken from WinDbg help):
-t[c|s] [Traces]
“Causes the debugger to display the collected traces of the heavy heap users. Traces specifies the number of traces to display; the default is four. If there are more traces than the specified number, the earliest traces are displayed. If -t or -tc is used, the traces are sorted by count usage. If -ts is used, the traces are sorted by size.”
September 8th, 2008 at 5:48 pm
[…] needed to check some data structures and how they change in the case of heap leaks and wrote a very small C program that was allocating memory in a loop using malloc function but […]
September 11th, 2008 at 4:11 pm
[…] Memory Leak (process heap) […]
January 13th, 2009 at 11:05 pm
[…] Memory profilers: Massif, AQtime and mpatrol (pp. 53 - 54) - On Windows you can use Gflags and select user mode stack trace database and then use WinDbg: http://www.dumpanalysis.org/blog/index.php/2007/08/06/crash-dump-analysis-patterns-part-20a/ […]
April 27th, 2009 at 11:22 am
[…] to take a few consecutive memory dumps of the growing memory and analyze it later as described in a heap leak pattern. This can also be a .NET leak too if unmanaged AppA.exe happened to load any managed […]
May 22nd, 2009 at 11:34 pm
[…] see similarities and differences. Most often this technique is used for memory leaks, for example, process heap leaks. Here we see another example related to CPU […]
August 4th, 2009 at 1:09 am
[…] was tracking down a memory leak using DMP files using a method outlined here. Windbg is great for this kind of memory leak debugging because you don’t have to worry about […]
October 6th, 2009 at 10:02 pm
[…] Apart from that, the size of the memory dump, almost 1.8Gb, suggested a memory leak and we clearly see expanded heaps that also suggest the case of a heap leak: […]
October 19th, 2009 at 9:10 pm
[…] I noticed yesterday that my home Vista computer suddenly became slower than usual so I brought Task Manager, sorted processes by CPU usage and discovered an instance of IE7 with 50% - 60% of CPU consumption. Dumping processes in Vista is easier than ever, so I did the right click on that process and selected Create Dump File menu option. The dump was saved and I killed the process. The size of the dump file was 1.2Gb and that definitely indicated a memory leak. Examining process heap showed large heap segments amounting to 800Mb and therefore pointing to the possible heap leak: […]
November 30th, 2009 at 4:47 pm
[…] all about deviations and of them is Size Deviation (a super pattern), be it a handle table size, a heap size, a number of contended locks, time spent in kernel, and so on. Every system or process property […]
May 12th, 2010 at 10:39 am
[…] we introduce an icon for Memory Leak (process heap) […]
May 3rd, 2013 at 10:01 am
We can get distribution stats for different block sizes and then filter all stack traces based on the specific size we are interested in:
0:001> !heap -stat -h 06a00000
heap @ 06a00000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
240 13eb9 - 31cce80 (40.67)
48 2a813 - bf4558 (9.76)
88 14213 - ab1a18 (8.73)
28 2a6a1 - 6a0928 (5.41)
4 187062 - 61c188 (4.99)
30 1e140 - 5a3c00 (4.61)
53218d 1 - 53218d (4.24)
50 fa85 - 4e4990 (4.00)
39d2d4 1 - 39d2d4 (2.95)
c 3592b - 282e04 (2.05)
3c 9591 - 230dfc (1.79)
b8 2b2b - 1f06e8 (1.58)
9c 31a9 - 1e42fc (1.54)
70 3592 - 176fe0 (1.20)
3ec 5dc - 16fad0 (1.17)
33c 5dc - 12f390 (0.97)
14 b4d5 - e20a4 (0.72)
20 6101 - c2020 (0.62)
6c 135e - 82ba8 (0.42)
60 1082 - 630c0 (0.32)
0:001> !heap -flt s 240
[…]
1c6c0db8 0055 0055 [00] 1c6c0dd0 00240 - (busy)
? ModuleA!DllUnregisterServer+272c7c
1c6c1060 0055 0055 [00] 1c6c1078 00240 - (busy)
? ModuleA!DllUnregisterServer+272c7c
1c6c1308 0055 0055 [00] 1c6c1320 00240 - (busy)
? ModuleA!DllUnregisterServer+272c7c
1c6c15b0 0055 0055 [00] 1c6c15c8 00240 - (busy)
? ModuleA!DllUnregisterServer+272c7c
1c6c1858 0055 0055 [00] 1c6c1870 00240 - (busy)
[…]
July 29th, 2017 at 6:43 am
There are 2 heap types in Windows 10: segment and NT
Two different heaps in Edge:
0:031> !heap
Heap Address NT/Segment Heap
2beae60000 Segment Heap
2beab10000 NT Heap
2beb4f0000 Segment Heap
2beadc0000 Segment Heap
2beae40000 Segment Heap
33ef7b0000 Segment Heap
33f15f0000 Segment Heap
2b88c50000 NT Heap
2ba2a50000 Segment Heap
2b9f940000 Segment Heap
2bac9f0000 Segment Heap
2bac9c0000 Segment Heap
2b9c9e0000 NT Heap
2b985d0000 NT Heap
2b823f0000 NT Heap
0:031> !heap -s
Process Total Total
Global Heap Reserved Committed
Heap Address Signature Flags List Bytes Bytes
Index (K) (K)
2beae60000 ddeeddee 0 1 94260 81524
2beb4f0000 ddeeddee 0 3 85044 69756
2beadc0000 ddeeddee 0 4 1076 80
2beae40000 ddeeddee 0 5 52 4
33ef7b0000 ddeeddee 0 6 1076 168
33f15f0000 ddeeddee 0 7 1076 36
2ba2a50000 ddeeddee 0 9 1076 144
2b9f940000 ddeeddee 0 10 1076 20
2bac9f0000 ddeeddee 0 11 3124 1856
2bac9c0000 ddeeddee 0 12 12340 8332
***********************************************************
NT HEAP STATS BELOW
***********************************************************
LFH Key : 0xd5b760accf32da62
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
————————————————————————————-
0000002beab10000 00008000 64 4 64 2 1 1 0 0
0000002b88c50000 00000001 16 16 16 13 1 1 0 N/A
0000002b9c9e0000 00000001 16 16 16 13 2 1 0 N/A
0000002b985d0000 00000001 16 16 16 13 1 1 0 N/A
0000002b823f0000 00000001 16 16 16 8 4 1 0 N/A
————————————————————————————-
https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals.pdf