Archive for August 12th, 2007

Visualizing Memory Leaks

Sunday, August 12th, 2007

Dump2Picture can be used to explore memory leaks visually. I created the following small program in Visual C++ that leaks 64Kb every second:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    printf("%x\n", (UINT_PTR)malloc(0xFFFF));
    Sleep(1000);
  }

  return 0;
}

Then I sampled 3 dumps at 7Mb, 17Mb and 32Mb process virtual memory size and converted them as 16 bits-per-pixel bitmaps. On the pictures below we can see that the middle black memory area grows significantly. Obviously malloc function allocates zeroed memory and therefore we see black color.

7Mb process memory dump:

17Mb process memory dump:

32Mb process memory dump:

If we zoom in the black area we would see the following pattern:

 

Colored lines inside are heap control structures that are created for every allocated block of memory. If this is correct then allocating smaller memory blocks would create a hatched pattern. And this is true indeed. The following program leaks 256 byte memory blocks:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    printf("%x\n", (UINT_PTR)malloc(0xFF));
    Sleep(1000/0xFF);
  }

  return 0;
}

The corresponding process memory picture and zoomed heap area are the following:

Making allocations 4 times smaller makes heap area to look hatched and zoomed picture is more densely packed by heap control structures:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    printf("%x\n", (UINT_PTR)malloc(0xFF/4));
    Sleep((1000/0xFF)/4);
  }

  return 0;
}

 

Here is another example. One service was increasing its memory constantly. The crash dump picture shows huge hatched dark region in the middle:

  

and if we zoom in this region:

Because the pattern and allocation size look uniform it could be the true heap memory leak for some operation that allocates constant size buffers. After opening the dump and looking at heap segments that had grown the most we see the same allocation size indeed:

0:000> !.\w2kfre\ntsdexts.heap -h 5
HEAPEXT: Unable to get address of NTDLL!NtGlobalFlag.
Index   Address  Name      Debugging options enabled
  1:   00140000
  2:   00240000
  3:   00310000
  4:   00330000
  5:   00370000
    Segment at 00370000 to 00380000 (00010000 bytes committed)
    Segment at 01680000 to 01780000 (00100000 bytes committed)
    Segment at 019C0000 to 01BC0000 (00200000 bytes committed)
    Segment at 01BC0000 to 01FC0000 (00400000 bytes committed)
    Segment at 01FC0000 to 027C0000 (00800000 bytes committed)
    Segment at 027C0000 to 037C0000 (01000000 bytes committed)
    Segment at 037C0000 to 057C0000 (02000000 bytes committed)

    Segment at 057C0000 to 097C0000 (00155000 bytes committed)



    057B96E0: 01048 . 01048 [07] - busy (1030), tail fill
    057BA728: 01048 . 01048 [07] - busy (1030), tail fill
    057BB770: 01048 . 01048 [07] - busy (1030), tail fill
    057BC7B8: 01048 . 01048 [07] - busy (1030), tail fill
    057BD800: 01048 . 01048 [07] - busy (1030), tail fill
    057BE848: 01048 . 01048 [07] - busy (1030), tail fill
    057BF890: 01048 . 00770 [14] free fill
  Heap entries for Segment07 in Heap 370000
    057C0040: 00040 . 01048 [07] - busy (1030), tail fill
    057C1088: 01048 . 01048 [07] - busy (1030), tail fill
    057C20D0: 01048 . 01048 [07] - busy (1030), tail fill
    057C3118: 01048 . 01048 [07] - busy (1030), tail fill
    057C4160: 01048 . 01048 [07] - busy (1030), tail fill
    057C51A8: 01048 . 01048 [07] - busy (1030), tail fill


- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 21)

Sunday, August 12th, 2007

Sometimes it is possible that a process crash dump doesn’t have all usual threads inside. For example, you expect at least 4 threads including the main process thread but in the dump you see only 3. If you know that some access violations were reported in the event log before (not necessarily for the same PID) you might suspect that one of threads had been terminated due to some reason. I call this pattern Missing Thread.

In order to simulate this problem I created a small multithreaded program in Visual C++:

#include "stdafx.h"
#include <process.h>

void thread_request(void *param)
{
    while (true);
}

int _tmain(int argc, _TCHAR* argv[])
{
    _beginthread(thread_request, 0, NULL);

    try
    {
        if (argc == 2)
        {
            *(int *)NULL = 0;
        }
    }
    catch (...)
    {
        _endthread();
    }

    while (true);

    return 0;
}

If there is a command line argument then the main thread simulates access violation and finishes in the exception handler. In order to use SEH exceptions with C++ try/catch blocks you have to enable /EHa option in C++ Code Generation properties:

If we run the program without command line parameter and take a manual dump from it we would see 2 threads:

0:000> ~*kL

.  0  Id: 1208.fdc Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0012ff70 00401403 MissingThread!wmain+0x58
0012ffc0 7d4e7d2a MissingThread!__tmainCRTStartup+0x15e
0012fff0 00000000 kernel32!BaseProcessStart+0x28

   1  Id: 1208.102c Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
005dff7c 004010ef MissingThread!thread_request
005dffb4 00401188 MissingThread!_callthreadstart+0x1b
005dffb8 7d4dfe21 MissingThread!_threadstart+0x73
005dffec 00000000 kernel32!BaseThreadStart+0x34

0:000> ~
.  0  Id: 1208.fdc Suspend: 1 Teb: 7efdd000 Unfrozen
   1  Id: 1208.102c Suspend: 1 Teb: 7efda000 Unfrozen

0:000> dd 7efdd000 l4
7efdd000  0012ff64 00130000 0012e000 00000000

I also dumped TEB of the main thread. However if we run the program with any command line parameter and look at its manual dump we would see only one thread with the main thread missing:

0:000> ~*kL

.  0  Id: 1004.12e8 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
005dff7c 004010ef MissingThread!thread_request
005dffb4 00401188 MissingThread!_callthreadstart+0x1b
005dffb8 7d4dfe21 MissingThread!_threadstart+0x73
005dffec 00000000 kernel32!BaseThreadStart+0x34

0:000> ~
.  0  Id: 1004.12e8 Suspend: 1 Teb: 7efda000 Unfrozen

If we try to dump TEB address and stack data from the missing main thread we would see that the memory was already decommitted:

0:000> dd 7efdd000 l4
7efdd000  ???????? ???????? ???????? ????????

0:000> dds 0012e000  00130000
0012e000  ????????
0012e004  ????????
0012e008  ????????
0012e00c  ????????
0012e010  ????????
0012e014  ????????
0012e018  ????????
0012e01c  ????????
0012e020  ????????
0012e024  ????????

The same effect can be achieved in the similar program that exits the thread in the custom unhandled exception filter:

#include "stdafx.h"
#include <process.h>
#include <windows.h>

LONG WINAPI CustomUnhandledExceptionFilter(struct _EXCEPTION_POINTERS* ExceptionInfo)
{
    ExitThread(-1);
}

void thread_request(void *param)
{
    while (true);
}

int _tmain(int argc, _TCHAR* argv[])
{
    _beginthread(thread_request, 0, NULL);
    SetUnhandledExceptionFilter(CustomUnhandledExceptionFilter);

    *(int *)NULL = 0;

    while (true);

    return 0;
}

The solution to catch an exception that results in a thread termination would be to run the program under WinDbg or any other debugger:

CommandLine: C:\MissingThread\MissingThread.exe 1
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 0040f000   MissingThread.exe
ModLoad: 7d4c0000 7d5f0000   NOT_AN_IMAGE
ModLoad: 7d600000 7d6f0000   C:\W2K3\SysWOW64\ntdll32.dll
ModLoad: 7d4c0000 7d5f0000   C:\W2K3\syswow64\kernel32.dll
(df0.12f0): Break instruction exception - code 80000003 (first chance)
eax=7d600000 ebx=7efde000 ecx=00000005 edx=00000020 esi=7d6a01f4 edi=00221f38
eip=7d61002d esp=0012fb4c ebp=0012fcac iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll32!DbgBreakPoint:
7d61002d cc              int     3

0:000> g
ModLoad: 71c20000 71c32000   C:\W2K3\SysWOW64\tsappcmp.dll
ModLoad: 77ba0000 77bfa000   C:\W2K3\syswow64\msvcrt.dll
ModLoad: 00410000 004ab000   C:\W2K3\syswow64\ADVAPI32.dll
ModLoad: 7da20000 7db00000   C:\W2K3\syswow64\RPCRT4.dll
ModLoad: 7d8d0000 7d920000   C:\W2K3\syswow64\Secur32.dll
(df0.12f0): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=000007a0 ebx=7d4d8df9 ecx=78b842d9 edx=00000000 esi=00000002 edi=00000ece
eip=00401057 esp=0012ff50 ebp=0012ff70 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
MissingThread!wmain+0x47:
00401057 c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

0:000> kL
ChildEBP RetAddr
0012ff70 00401403 MissingThread!wmain+0x47
0012ffc0 7d4e7d2a MissingThread!__tmainCRTStartup+0x15e
0012fff0 00000000 kernel32!BaseProcessStart+0x28

If live debugging is not possible and you are interested in crash dumps saved upon a first chance exception before it is processed in an exception handler you can also use MS userdump after you install it and enable All Exceptions in the Process Monitoring Rules dialog box. Another tool can be used is ADPlus in crash mode from Debugging Tools for Windows.

- Dmitry Vostokov @ DumpAnalysis.org -