Crash Dump Analysis Patterns (Part 20a)

Memory Leak is another pattern that may be finally manifested as Insufficient Memory pattern in a crash dump. In this part I’ll cover process heap memory leaks. They are usually identified when the process virtual memory size grows over time. It starts with 80Mb and instead of fluctuating normally below 100Mb it suddenly starts growing to 150Mb after some time and then to 300Mb next day and then grows to 600Mb and so on.

Usually a process heap is under suspicion here. To confirm this we need to sample 2-3 consecutive user memory dumps at process sizes 100Mb, 200Mb and 300Mb, for example. This can be done by using Microsoft userdump.exe command line tool. Then we can see whether there is any heap growth by using !heap -s WinDbg command:

1st dump

0:000> !heap -s
  Heap     Flags   Reserv  Commit  Virt
                    (k)     (k)    (k)
---------------------------------------
00140000 00000002    2048   1048   1112
00240000 00008000      64     12     12
00310000 00001002    7232   4308   4600
00420000 00001002    1024    520    520
00340000 00001002     256     40     40
00720000 00001002      64     32     32
00760000 00001002      64     48     48
01020000 00001002     256     24     24
02060000 00001002      64     16     16
02070000 00001003     256    120    120
020b0000 00001003     256      4      4
020f0000 00001003     256      4      4
02130000 00001003     256      4      4
02170000 00001003     256      4      4
021f0000 00001002    1088     76     76
021e0000 00001002      64     16     16
02330000 00001002    1088    428    428
02340000 00011002     256     12     12
02380000 00001002      64     12     12
024c0000 00001003      64      8      8
028d0000 00001002    7232   3756   6188
02ce0000 00001003      64      8      8
07710000 00001002      64     20     20
07b20000 00001002      64     16     16
07f30000 00001002      64     16     16
09050000 00001002     256     12     12
09c80000 00001002  130304 102340 102684
007d0000 00001003     256    192    192
00810000 00001003     256      4      4
0bdd0000 00001003     256      4      4
0be10000 00001003     256      4      4
0be50000 00001003     256      4      4
0be90000 00001003     256     56     56
0bed0000 00001003     256      4      4
0bf10000 00001003     256      4      4
0bf50000 00001003     256      4      4
0bf90000 00001003     256      4      4
00860000 00001002      64     20     20
00870000 00001002      64     20     20
0d760000 00001002     256     12     12
0dc60000 00001002    1088    220    220
0c3a0000 00001002      64     12     12
0c3d0000 00001002    1088    160    364
08420000 00001002      64     64     64

2nd dump

0:000> !heap -s
  Heap     Flags   Reserv  Commit  Virt
                    (k)     (k)    (k)
---------------------------------------
00140000 00000002    8192   4600   4600
00240000 00008000      64     12     12
00310000 00001002    7232   4516   4600
00420000 00001002    1024    520    520
00340000 00001002     256     44     44
00720000 00001002      64     32     32
00760000 00001002      64     48     48
01020000 00001002     256     24     24
02060000 00001002      64     16     16
02070000 00001003     256    124    124
020b0000 00001003     256      4      4
020f0000 00001003     256      4      4
02130000 00001003     256      4      4
02170000 00001003     256      4      4
021f0000 00001002    1088     76     76
021e0000 00001002      64     16     16
02330000 00001002    1088    428    428
02340000 00011002     256     12     12
02380000 00001002      64     12     12
024c0000 00001003      64      8      8
028d0000 00001002    7232   3796   6768
02ce0000 00001003      64      8      8
07710000 00001002      64     20     20
07b20000 00001002      64     16     16
07f30000 00001002      64     16     16
09050000 00001002     256     12     12
09c80000 00001002  261376 221152 221928
007d0000 00001003     256    192    192
00810000 00001003     256      4      4
0bdd0000 00001003     256      4      4
0be10000 00001003     256      4      4
0be50000 00001003     256      4      4
0be90000 00001003     256     60     60
0bed0000 00001003     256      4      4
0bf10000 00001003     256      4      4
0bf50000 00001003     256      4      4
0bf90000 00001003     256      4      4
00860000 00001002      64     20     20
00870000 00001002      64     20     20
0d760000 00001002     256     12     12
0dc60000 00001002    1088    228    228
0c3a0000 00001002      64     12     12
0c3d0000 00001002    1088    168    224
08450000 00001002      64     64     64

We see that the only significant heap growth is at 09c80000 address, from 130Mb to 260Mb. However this doesn’t say which code uses it. In order to find the code we need to enable the so called user mode stack trace database. Please refer to the following Citrix article:

http://support.citrix.com/article/CTX106970

The example in the article is for Citrix IMA service but you can replace ImaSrv.exe with any other executable name.

Suppose that after enabling user mode stack trace database and restarting the program or service we see the growth and we get memory dumps with the following suspicious heap highlighted in red:

0:000> !gflag
Current NtGlobalFlag contents: 0x00001000
ust - Create user mode stack trace database

0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
  stack back traces
LFH Key: 0x2687ed29
  Heap     Flags   Reserv  Commit  Virt
                    (k)     (k)    (k)
---------------------------------------
00140000 58000062    4096    488    676
00240000 58008060      64     12     12
00360000 58001062    3136   1152   1216
003b0000 58001062      64     32     32
01690000 58001062     256     32     32
016d0000 58001062    1024    520    520
003e0000 58001062      64     48     48
02310000 58001062     256     24     24
02b30000 58001062      64     16     16
02b40000 58001063     256     64     64
02b80000 58001063     256      4      4
02bc0000 58001063     256      4      4
02c00000 58001063     256      4      4
02c40000 58001063     256      4      4
02c80000 58001063     256      4      4
02cc0000 58001063     256      4      4
02d30000 58001063      64      4      4
03140000 58001062    7232   4160   4896
03550000 58001063      64      4      4
07f70000 58001062      64     12     12
08380000 58001062      64     12     12
08790000 58001062      64     12     12
091d0000 58011062     256     12     12
09210000 58001062      64     16     16
09220000 58001062      64     12     12
092a0000 58001062      64     12     12
09740000 58001062     256     12     12
0b1a0000 58001062      64     12     12
0b670000 58001062   64768  39508  39700
0b7b0000 58001062      64     12     12
0c650000 58001062    1088    192    192

Every heap is subdivided into several segments and to see which segments have grown the most we can use !heap -m <heap address> command:

0:000> !heap -m 0b670000
Index   Address  Name      Debugging options enabled
29:   0b670000
    Segment at 0b670000 to 0b6b0000 (00040000 bytes committed)
    Segment at 0c760000 to 0c860000 (00100000 bytes committed)
    Segment at 0c980000 to 0cb80000 (001fe000 bytes committed)
    Segment at 0cb80000 to 0cf80000 (003cc000 bytes committed)
    Segment at 0dc30000 to 0e430000 (00800000 bytes committed)
    Segment at 12330000 to 13330000 (01000000 bytes committed)
    Segment at 13330000 to 15330000 (0078b000 bytes committed)



If we use !heap -a <heap address> command then in addition to the list of heap segments individual heap allocation entries will be dumped as well. This could be very big output and we should open the log file in advance by using .logopen <file name> command.

The output can be like this (taken from another dump):

0:000> !heap -a 000a0000
...
...
...
    Segment00 at 000a0000:
        Flags:           00000000
        Base:            000a0000
        First Entry:     000a0580
        Last Entry:      000b0000
        Total Pages:     00000010
        Total UnCommit:  00000002
        Largest UnCommit:00000000
        UnCommitted Ranges: (1)

    Heap entries for Segment00 in Heap 000a0000
        000a0000: 00000 . 00580 [101] - busy (57f)
        000a0580: 00580 . 00240 [101] - busy (23f)
        000a07c0: 00240 . 00248 [101] - busy (22c)
        000a0a08: 00248 . 00218 [101] - busy (200)
        000a0c20: 00218 . 00ce0 [100]
        000a1900: 00ce0 . 00f88 [101] - busy (f6a)
        000a2888: 00f88 . 04418 [101] - busy (4400)
        000a6ca0: 04418 . 05958 [101] - busy (5940)
        000ac5f8: 05958 . 00928 [101] - busy (90c)
        000acf20: 00928 . 010c0 [100]
        000adfe0: 010c0 . 00020 [111] - busy (1d)
        000ae000:      00002000      - uncommitted bytes.

Then we can inspect individual entries to see stack traces that allocated them by using !heap -p -a <heap entry address> command:

0:000> !heap -p -a 000a6ca0
    address 000a6ca0 found in
    _HEAP @ a0000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        000a6ca0 0b2b 0000  [00]   000a6cb8    05940 - (busy)
        Trace: 2156ac
        7704dab4 ntdll!RtlAllocateHeap+0x0000021d
        75c59b12 USP10!UspAllocCache+0x0000002b
        75c62381 USP10!AllocSizeCache+0x00000048
        75c61c74 USP10!FindOrCreateSizeCacheWithoutRealizationID+0x00000124
        75c61bc0 USP10!FindOrCreateSizeCacheUsingRealizationID+0x00000070
        75c59a97 USP10!UpdateCache+0x0000002b
        75c59a61 USP10!ScriptCheckCache+0x0000005c
        75c59d04 USP10!ScriptStringAnalyse+0x0000012a
        7711140f LPK!LpkStringAnalyse+0x00000114
        7711159e LPK!LpkCharsetDraw+0x00000302
        77111488 LPK!LpkDrawTextEx+0x00000044
        76a4beb3 USER32!DT_DrawStr+0x0000013a
        76a4be45 USER32!DT_DrawJustifiedLine+0x0000005f
        76a49d68 USER32!AddEllipsisAndDrawLine+0x00000186
        76a4bc31 USER32!DrawTextExWorker+0x000001b1
        76a4bedc USER32!DrawTextExW+0x0000001e
        746051d8 uxtheme!CTextDraw::GetTextExtent+0x000000be
        7460515a uxtheme!GetThemeTextExtent+0x00000065
        74611ed4 uxtheme!CThemeMenuBar::MeasureItem+0x00000124
        746119c1 uxtheme!CThemeMenu::OnMeasureItem+0x0000003f
        74611978 uxtheme!CThemeWnd::_PreDefWindowProc+0x00000117
        74601ea5 uxtheme!_ThemeDefWindowProc+0x00000090
        74601f61 uxtheme!ThemeDefWindowProcW+0x00000018
        76a4a09e USER32!DefWindowProcW+0x00000068
        931406 notepad!NPWndProc+0x00000084
        76a51a10 USER32!InternalCallWinProc+0x00000023
        76a51ae8 USER32!UserCallWinProcCheckWow+0x0000014b
        76a51c03 USER32!DispatchClientMessage+0x000000da
        76a3bc24 USER32!__fnINOUTLPUAHMEASUREMENUITEM+0x00000027
        77040e6e ntdll!KiUserCallbackDispatcher+0x0000002e
        76a51d87 USER32!RealDefWindowProcW+0x00000047
        74601f2f uxtheme!_ThemeDefWindowProc+0x000001b8

If we want to dump all heap entries with their corresponding stack traces we can use !heap -k -h <heap address> command.

Note: sometimes all these commands don’t work. In such cases we can use old Windows 2000 extension.

Some prefer to use umdh.exe and get text file logs but the advantage of embedding heap allocation stack traces in a dump is that we are not concerned with sending and configuring symbol files at a customer side.

- Dmitry Vostokov @ DumpAnalysis.org -

15 Responses to “Crash Dump Analysis Patterns (Part 20a)”

  1. Tal Rosen Says:

    You can also use Microsoft Debug Diagnostics Tool.
    Sometimes it manages to pinpoint the leaking DLL, even without symbols.

    see http://blogs.msdn.com/debugdiag/

  2. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 20b) Says:

    […] the process size constantly grows but there is no difference in the process heap size. In such cases we need to check whether the process uses Microsoft .NET runtime (CLR). If one of […]

  3. Dmitry Vostokov Says:

    various pageheap options !heap -p are useful such as (taken from WinDbg help):

    -t[c|s] [Traces]

    “Causes the debugger to display the collected traces of the heavy heap users. Traces specifies the number of traces to display; the default is four. If there are more traces than the specified number, the earliest traces are displayed. If -t or -tc is used, the traces are sorted by count usage. If -ts is used, the traces are sorted by size.”

  4. Crash Dump Analysis » Blog Archive » Large Heap Allocations Says:

    […] needed to check some data structures and how they change in the case of heap leaks and wrote a very small C program that was allocating memory in a loop using malloc function but […]

  5. Crash Dump Analysis » Blog Archive » First-order and second-order memory leaks Says:

    […] Memory Leak (process heap) […]

  6. Software Generalist » Blog Archive » Reading Notebook: 13-Jan-09 Says:

    […] Memory profilers: Massif, AQtime and mpatrol (pp. 53 - 54) - On Windows you can use Gflags and select user mode stack trace database and then use WinDbg: http://www.dumpanalysis.org/blog/index.php/2007/08/06/crash-dump-analysis-patterns-part-20a/ […]

  7. Crash Dump Analysis » Blog Archive » Looking for abnormal: case study Says:

    […] to take a few consecutive memory dumps of the growing memory and analyze it later as described in a heap leak pattern. This can also be a .NET leak too if unmanaged AppA.exe happened to load any managed […]

  8. Crash Dump Analysis » Blog Archive » Comparative Memory Dump Analysis: CPU Spikes Says:

    […] see similarities and differences. Most often this technique is used for memory leaks, for example, process heap leaks. Here we see another example related to CPU […]

  9. Say Goodnight » WTF? “tail fill – unable to read heap entry extra” Says:

    […] was tracking down a memory leak using DMP files using a method outlined here. Windbg is great for this kind of memory leak debugging because you don’t have to worry about […]

  10. Crash Dump Analysis » Blog Archive » Stack trace collection, message box, hidden exception, nested offender, insufficient memory, C++ exception, heap leak and ubiquitous component: pattern cooperation Says:

    […] Apart from that, the size of the memory dump, almost 1.8Gb, suggested a memory leak and we clearly see expanded heaps that also suggest the case of a heap leak: […]

  11. Crash Dump Analysis » Blog Archive » Memory leak, spiking threads, wait chain, high critical section contention and module variety: pattern cooperation Says:

    […] I noticed yesterday that my home Vista computer suddenly became slower than usual so I brought Task Manager, sorted processes by CPU usage and discovered an instance of IE7 with 50% - 60% of CPU consumption. Dumping processes in Vista is easier than ever, so I did the right click on that process and selected Create Dump File menu option. The dump was saved and I killed the process. The size of the dump file was 1.2Gb and that definitely indicated a memory leak. Examining process heap showed large heap segments amounting to 800Mb and therefore pointing to the possible heap leak: […]

  12. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 94a) Says:

    […] all about deviations and of them is Size Deviation (a super pattern), be it a handle table size, a heap size, a  number of contended locks, time spent in kernel, and so on. Every system or process property […]

  13. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 37) Says:

    […] we introduce an icon for Memory Leak (process heap) […]

  14. Dmitry Vostokov Says:

    We can get distribution stats for different block sizes and then filter all stack traces based on the specific size we are interested in:

    0:001> !heap -stat -h 06a00000
    heap @ 06a00000
    group-by: TOTSIZE max-display: 20
    size #blocks total ( %) (percent of total busy bytes)
    240 13eb9 - 31cce80 (40.67)
    48 2a813 - bf4558 (9.76)
    88 14213 - ab1a18 (8.73)
    28 2a6a1 - 6a0928 (5.41)
    4 187062 - 61c188 (4.99)
    30 1e140 - 5a3c00 (4.61)
    53218d 1 - 53218d (4.24)
    50 fa85 - 4e4990 (4.00)
    39d2d4 1 - 39d2d4 (2.95)
    c 3592b - 282e04 (2.05)
    3c 9591 - 230dfc (1.79)
    b8 2b2b - 1f06e8 (1.58)
    9c 31a9 - 1e42fc (1.54)
    70 3592 - 176fe0 (1.20)
    3ec 5dc - 16fad0 (1.17)
    33c 5dc - 12f390 (0.97)
    14 b4d5 - e20a4 (0.72)
    20 6101 - c2020 (0.62)
    6c 135e - 82ba8 (0.42)
    60 1082 - 630c0 (0.32)

    0:001> !heap -flt s 240
    […]
    1c6c0db8 0055 0055 [00] 1c6c0dd0 00240 - (busy)
    ? ModuleA!DllUnregisterServer+272c7c
    1c6c1060 0055 0055 [00] 1c6c1078 00240 - (busy)
    ? ModuleA!DllUnregisterServer+272c7c
    1c6c1308 0055 0055 [00] 1c6c1320 00240 - (busy)
    ? ModuleA!DllUnregisterServer+272c7c
    1c6c15b0 0055 0055 [00] 1c6c15c8 00240 - (busy)
    ? ModuleA!DllUnregisterServer+272c7c
    1c6c1858 0055 0055 [00] 1c6c1870 00240 - (busy)
    […]

  15. Dmitry Vostokov Says:

    There are 2 heap types in Windows 10: segment and NT

    Two different heaps in Edge:

    0:031> !heap
    Heap Address NT/Segment Heap

    2beae60000 Segment Heap
    2beab10000 NT Heap
    2beb4f0000 Segment Heap
    2beadc0000 Segment Heap
    2beae40000 Segment Heap
    33ef7b0000 Segment Heap
    33f15f0000 Segment Heap
    2b88c50000 NT Heap
    2ba2a50000 Segment Heap
    2b9f940000 Segment Heap
    2bac9f0000 Segment Heap
    2bac9c0000 Segment Heap
    2b9c9e0000 NT Heap
    2b985d0000 NT Heap
    2b823f0000 NT Heap

    0:031> !heap -s

    Process Total Total
    Global Heap Reserved Committed
    Heap Address Signature Flags List Bytes Bytes
    Index (K) (K)

    2beae60000 ddeeddee 0 1 94260 81524
    2beb4f0000 ddeeddee 0 3 85044 69756
    2beadc0000 ddeeddee 0 4 1076 80
    2beae40000 ddeeddee 0 5 52 4
    33ef7b0000 ddeeddee 0 6 1076 168
    33f15f0000 ddeeddee 0 7 1076 36
    2ba2a50000 ddeeddee 0 9 1076 144
    2b9f940000 ddeeddee 0 10 1076 20
    2bac9f0000 ddeeddee 0 11 3124 1856
    2bac9c0000 ddeeddee 0 12 12340 8332

    ***********************************************************
    NT HEAP STATS BELOW
    ***********************************************************
    LFH Key : 0xd5b760accf32da62
    Termination on corruption : ENABLED
    Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
    (k) (k) (k) (k) length blocks cont. heap
    ————————————————————————————-
    0000002beab10000 00008000 64 4 64 2 1 1 0 0
    0000002b88c50000 00000001 16 16 16 13 1 1 0 N/A
    0000002b9c9e0000 00000001 16 16 16 13 2 1 0 N/A
    0000002b985d0000 00000001 16 16 16 13 1 1 0 N/A
    0000002b823f0000 00000001 16 16 16 8 4 1 0 N/A
    ————————————————————————————-

    https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals.pdf

Leave a Reply