Crash Dump Analysis

CARE: Crash Analysis Report Environment

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

I needed to check some data structures and how they change in the case of heap memory leaks and wrote a very small C program that was allocating memory in a loop using malloc function. The VM size was growing very fast and I saved process memory dumps at 200Mb and 500Mb. When checking heap segments I noticed that they had not increased although the process was allocating 0×1000000 chunks of heap memory:

0:000> !heap 0 0
Index   Address  Name      Debugging options enabled
  1:   00260000
    Segment at 0000000000260000 to 0000000000360000 (00008000 bytes committed)
  2:   00360000
    Segment at 0000000000360000 to 0000000000370000 (00004000 bytes committed)
  3:   00440000
    Segment at 0000000000440000 to 0000000000450000 (00010000 bytes committed)
    Segment at 0000000000450000 to 0000000000550000 (00021000 bytes committed)
  4:   00560000
    Segment at 0000000000560000 to 0000000000570000 (00010000 bytes committed)
    Segment at 0000000000570000 to 0000000000670000 (0003a000 bytes committed)

I was puzzled because inspection of virtual memory showed those chunks as belonging to heap regions:

0:000> !address
[...]
    0000000009700000 : 0000000009700000 - 0000000001002000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   0000000000560000
    000000000a702000 : 000000000a702000 - 000000000000e000
                    Type     00000000
                    Protect  00000001 PAGE_NOACCESS
                    State    00010000 MEM_FREE
                    Usage    RegionUsageFree
    000000000a710000 : 000000000a710000 - 0000000001002000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   0000000000560000
    000000000b712000 : 000000000b712000 - 0000000004aee000
                    Type     00000000
                    Protect  00000001 PAGE_NOACCESS
                    State    00010000 MEM_FREE
                    Usage    RegionUsageFree
[…] 

And then I recalled that large allocations for a process heap go to a separate linked list:

0:000> !peb
PEB at 000007fffffdb000

0:000> dt _PEB 000007fffffdb000
ntdll!_PEB
[...]
   +0x0f0 ProcessHeaps     : 0×00000000`77fa3460  -> 0×00000000`00260000
[..]

0:000> dq 0×00000000`77fa3460
00000000`77fa3460  00000000`00260000 00000000`00360000
00000000`77fa3470  00000000`00440000 00000000`00560000
00000000`77fa3480  00000000`00000000 00000000`00000000
00000000`77fa3490  00000000`00000000 00000000`00000000
00000000`77fa34a0  00000000`00000000 00000000`00000000
00000000`77fa34b0  00000000`00000000 00000000`00000000
00000000`77fa34c0  00000000`00000000 00000000`00000000
00000000`77fa34d0  00000000`00000000 00000000`00000000

0:000> dt _HEAP 00000000`00260000
ntdll!_HEAP
[...]
   +0×090 VirtualAllocdBlocks : _LIST_ENTRY [ 0×00000000`00260090 - 0×260090 ]
[…]

0:000> dl 00000000`00260000+90 10 2
00000000`00260090  00000000`00260090 00000000`00260090

0:000> dl 00000000`00360000+90 10 2
00000000`00360090  00000000`00360090 00000000`00360090

0:000> dl 00000000`00440000+90 10 2
00000000`00440090  00000000`00440090 00000000`00440090

0:000> dl 00000000`00560000+90 10 2
00000000`00560090  00000000`00670000 00000000`0a710000
00000000`00670000  00000000`01680000 00000000`00560090
00000000`01680000  00000000`02690000 00000000`00670000
00000000`02690000  00000000`036a0000 00000000`01680000
00000000`036a0000  00000000`046b0000 00000000`02690000
00000000`046b0000  00000000`056c0000 00000000`036a0000
00000000`056c0000  00000000`066d0000 00000000`046b0000
00000000`066d0000  00000000`076e0000 00000000`056c0000
00000000`076e0000  00000000`086f0000 00000000`066d0000
00000000`086f0000  00000000`09700000 00000000`076e0000
00000000`09700000  00000000`0a710000 00000000`086f0000
00000000`0a710000  00000000`00560090 00000000`09700000

We see that the last process heap has large allocations directly from virtual memory, for example:

0:000> !address 00000000`0a710000
    000000000a710000 : 000000000a710000 - 0000000001002000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   0000000000560000

Actually if I used heap statistics option for !heap command I would see these large allocations:

0:000> !heap -s
LFH Key                   : 0x000000a4e8aa078c
          Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                            (k)     (k)    (k)     (k) length      blocks cont. heap
0000000000260000 00000002    1024     32     32      7     1     1    0      0   L 
0000000000360000 00008000      64     16     16     12     1     1    0      0     
0000000000440000 00001002    1088    196    196      4     1     1    0      0   LFH
Virtual block: 0000000000670000 - 0000000000670000
Virtual block: 0000000001680000 - 0000000001680000
Virtual block: 0000000002690000 - 0000000002690000
Virtual block: 00000000036a0000 - 00000000036a0000
Virtual block: 00000000046b0000 - 00000000046b0000
Virtual block: 00000000056c0000 - 00000000056c0000
Virtual block: 00000000066d0000 - 00000000066d0000
Virtual block: 00000000076e0000 - 00000000076e0000
Virtual block: 00000000086f0000 - 00000000086f0000
Virtual block: 0000000009700000 - 0000000009700000
Virtual block: 000000000a710000 - 000000000a710000
0000000000560000 00001002    1088    296    296     18     3     1   11      0   LFH


The dump file can be downloaded from FTP to play with:

ftp://dumpanalysis.org/pub/LargeHeapAllocations.zip 

- Dmitry Vostokov @ DumpAnalysis.org -

           

Announcements

Coming Soon:

Debugging Notebook: Essential Concepts, WinDbg Commands and Tools

Crash Dump Analysis for System Administrators and Support Engineers

New Magazines:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers

New Books:

Memory Dump Analysis Anthology, Volume 3

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

x64 Windows Debugging: Practical Foundations

Also available:

Windows Debugging: Practical Foundations

DLL List Landscape: The Art from Computer Memory Space

Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

WinDbg: A Reference Poster and Learning Cards

Memory Dump Analysis Anthology, Volume 2

Memory Dump Analysis Anthology, Volume 1

New Children's Book:

Baby Turing

This entry was posted on Monday, September 8th, 2008 at 5:48 pm and is filed under Crash Dump Analysis, Debugging, WinDbg Tips and Tricks. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.