Crash Dump Analysis Patterns (Part 23a)

Double-free bugs lead to Dynamic Memory Corruption. The reason why Double Free deserves its own pattern name is the fact that either debug runtime libraries or even OS itself detect such bugs and save crash dumps immediately.

For some heap implementations double free doesn’t lead to an immediate heap corruption and subsequent crash. For example, if you allocate 3 blocks in a row and then free the middle one twice there will be no crash as the second free call is able to detect that the block was already freed and does nothing. The following program loops forever and never crashes:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    puts("Allocate: p1");
    void *p1 = malloc(100);
    puts("Allocate: p2");
    void *p2 = malloc(100);
    puts("Allocate: p3");
    void *p3 = malloc(100);

    puts("Free: p2");
    free(p2);
    puts(”Double-Free: p2″);
    free(p2);

    puts(”Free: p1″);
    free(p1);
    puts(”Free: p3″);
    free(p3);

    Sleep(100);
  }

  return 0;
}

The output of the program: 

...
...
...
Allocate: p1
Allocate: p2
Allocate: p3
Free: p2
Double-Free: p2
Free: p1
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p2
Double-Free: p2
Free: p1
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p2
Double-Free: p2
...
...
...

However if a free call triggered heap coalescence (adjacent free blocks form the bigger free block) then we have a heap corruption crash on the next double-free call because the coalescence triggered by the previous free call erased free block information:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    puts("Allocate: p1");
    void *p1 = malloc(100);
    puts("Allocate: p2");
    void *p2 = malloc(100);
    puts("Allocate: p3");
    void *p3 = malloc(100);

    puts("Free: p3");
    free(p3);
    puts("Free: p1");
    free(p1);
    puts("Free: p2");
    free(p2);
    puts(”Double-Free: p2″);
    free(p2);

    Sleep(100);
  }

  return 0;
}

The output of the program:

Allocate: p1
Allocate: p2
Allocate: p3
Free: p3
Free: p1
Free: p2
Double-Free: p2
Crash!

If we open a crash dump we would see the following stack trace:

0:000> r
eax=00922130 ebx=00920000 ecx=10101010 edx=10101010 esi=00922128 edi=00921fc8
eip=76ee1ad5 esp=0012fd6c ebp=0012fd94 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
ntdll!RtlpCoalesceFreeBlocks+0x6ef:
76ee1ad5 8b4904          mov     ecx,dword ptr [ecx+4] ds:0023:10101014=????????

0:000> kL
ChildEBP RetAddr
0012fd94 76ee1d37 ntdll!RtlpCoalesceFreeBlocks+0x6ef
0012fe8c 76ee1c21 ntdll!RtlpFreeHeap+0x1e2
0012fea8 758d7a7e ntdll!RtlFreeHeap+0x14e
0012febc 6cff4c39 kernel32!HeapFree+0x14
0012ff08 0040107b msvcr80!free+0xcd
0012ff5c 004011f1 DoubleFree!wmain+0x7b
0012ffa0 758d3833 DoubleFree!__tmainCRTStartup+0x10f
0012ffac 76eba9bd kernel32!BaseThreadInitThunk+0xe
0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23

This is illustrated on the following picture where free calls result in heap coalescence and the subsequent double-free call corrupts the heap:

The problem here is that heap coalescence can be triggered some time after the double free so we need some solution to diagnose double-free bugs earlier, ideally at the first double-free call. For example, the following program crashes during the normal free operation long after the first double-free happened:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    puts("Allocate: p1");
    void *p1 = malloc(100);
    puts("Allocate: p2");
    void *p2 = malloc(100);
    puts("Allocate: p3");
    void *p3 = malloc(100);

    puts("Free: p1");
    free(p1);
    puts("Free: p2");
    free(p2);
    puts(”Double-Free: p2″);
    free(p2);

    puts(”Double-Free: p3″);
    free(p3);

    Sleep(100);
  }

  return 0;
}

The output of the program:

Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Crash!

If we enable full page heap using gflags.exe from Debugging Tools for Windows the program crashes immediately on the double free call:

Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Crash!

The crash dump shows the following stack trace:

0:000> kL
ChildEBP RetAddr
0012f810 71aa4ced ntdll!DbgBreakPoint+0x1
0012f834 71aa9fc2 verifier!VerifierStopMessage+0x1fd
0012f890 71aaa4da verifier!AVrfpDphReportCorruptedBlock+0x102
0012f8a4 71ab2c98 verifier!AVrfpDphCheckNormalHeapBlock+0x18a
0012f8b8 71ab2a0e verifier!_EH4_CallFilterFunc+0x12
0012f8e0 76ee1039 verifier!_except_handler4+0x8e
0012f904 76ee100b ntdll!ExecuteHandler2+0x26
0012f9ac 76ee0e97 ntdll!ExecuteHandler+0x24
0012f9ac 71aaa3ad ntdll!KiUserExceptionDispatcher+0xf
0012fcf0 71aaa920 verifier!AVrfpDphCheckNormalHeapBlock+0x5d
0012fd0c 71aa879b verifier!AVrfpDphNormalHeapFree+0x20
0012fd60 76f31c8f verifier!AVrfDebugPageHeapFree+0x1cb
0012fda8 76efd9fa ntdll!RtlDebugFreeHeap+0x2f
0012fe9c 76ee1c21 ntdll!RtlpFreeHeap+0x5f
0012feb8 758d7a7e ntdll!RtlFreeHeap+0x14e
0012fecc 6cff4c39 kernel32!HeapFree+0x14
0012ff18 0040105f msvcr80!free+0xcd
0012ff5c 004011f1 DoubleFree!wmain+0x5f
0012ffa0 758d3833 DoubleFree!__tmainCRTStartup+0x10f
0012ffac 76eba9bd kernel32!BaseThreadInitThunk+0xe

0:000> !gflag
Current NtGlobalFlag contents: 0x02000000
    hpa - Place heap allocations at ends of pages

If we enable heap free checking instead of page heap we get our crash on the first double free call immediately too:

Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Crash!

The crash dump shows the following stack trace:

0:000> r
eax=feeefeee ebx=001b2040 ecx=001b0000 edx=001b2040 esi=d4476047 edi=001b2038
eip=76ee2086 esp=0012fe68 ebp=0012fe9c iopl=0 nv up ei ng nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010286
ntdll!RtlpLowFragHeapFree+0x31:
76ee2086 8b4604          mov     eax,dword ptr [esi+4] ds:0023:d447604b=????????

0:000> kL
ChildEBP RetAddr
0012fe9c 76ee18c3 ntdll!RtlpLowFragHeapFree+0x31
0012feb0 758d7a7e ntdll!RtlFreeHeap+0x101
0012fec4 6cff4c39 kernel32!HeapFree+0x14
0012ff10 0040106d msvcr80!free+0xcd
0012ff5c 004011f1 DoubleFree!wmain+0x6d
0012ffa0 758d3833 DoubleFree!__tmainCRTStartup+0x10f
0012ffac 76eba9bd kernel32!BaseThreadInitThunk+0xe
0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23

0:000> !gflag
Current NtGlobalFlag contents: 0x00000020
    hfc - Enable heap free checking

- Dmitry Vostokov @ DumpAnalysis.org -

6 Responses to “Crash Dump Analysis Patterns (Part 23a)”

  1. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 71) Says:

    […] happen due to corrupt or overwritten heap or pool control structures (for the latter see Double Free pattern). Another frequently seen specialization is called Critical Section Corruption which is […]

  2. Nitin Says:

    Hi,
    I tried all the examples given in this page which are supposed to genereate a crash, with pageheap enabled. But I could not see any crash. I was running the programs with and without debugger. OS was w2k3 and w2k8.

  3. Crash Dump Analysis » Blog Archive » Crash Dump Analysis AntiPatterns (Part 13) Says:

    […] to catch buffer overwrites but not underwrites and heap can also be damaged by other means like double free, passing an invalid address or direct corruption of control structures via a dangling pointer. […]

  4. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 41) Says:

    […] we introduce an icon for Double Free (process heap) […]

  5. Dmitry Vostokov Says:

    An example of Double Free detected in Windows 7:

    0:048> k
    ChildEBP RetAddr
    206dee98 777f8567 ntdll!ZwWaitForSingleObject+0×15
    206def1c 777f8695 ntdll!RtlReportExceptionEx+0×14b
    206def74 7781e6e6 ntdll!RtlReportException+0×86
    206def88 7781e763 ntdll!RtlpTerminateFailureFilter+0×14
    206def94 777c73dc ntdll!RtlReportCriticalFailure+0×67
    206defa8 777c7281 ntdll!_EH4_CallFilterFunc+0×12
    206defd0 777ab499 ntdll!_except_handler4+0×8e
    206deff4 777ab46b ntdll!ExecuteHandler2+0×26
    206df018 777ab40e ntdll!ExecuteHandler+0×24
    206df0a4 77760133 ntdll!RtlDispatchException+0×127
    206df0a4 7781e753 ntdll!KiUserExceptionDispatcher+0xf
    206df5e8 7781f659 ntdll!RtlReportCriticalFailure+0×57
    206df5f8 7781f739 ntdll!RtlpReportHeapFailure+0×21
    206df62c 777ce045 ntdll!RtlpLogHeapFailure+0xa1
    206df65c 76aa6e6a ntdll!RtlFreeHeap+0×64
    206df670 58110076 ole32!CRetailMalloc_Free+0×1c [d:\w7rtm\com\ole32\com\class\memapi.cxx @ 687]
    WARNING: Stack unwind information not available. Following frames may be wrong.
    206df6ac 581100e9 OUTLMIME!MimeOleInetDateToFileTime+0xd562
    206df6b8 5811051d OUTLMIME!MimeOleInetDateToFileTime+0xd5d5
    206df6e0 771562fa OUTLMIME!MimeOleInetDateToFileTime+0xda09
    206df70c 77156d3a user32!InternalCallWinProc+0×23
    206df784 771577c4 user32!UserCallWinProcCheckWow+0×109
    206df7e4 77157bca user32!DispatchMessageWorker+0×3bc
    206df7f4 581d74e6 user32!DispatchMessageA+0xf
    206df830 581e04a3 OUTLPH!DllGetClassObject+0×5616
    206df84c 581df9ac OUTLPH!DllGetClassObject+0xe5d3
    206df880 6e558488 OUTLPH!DllGetClassObject+0xdadc
    206df8a4 650fa17d OLMAPI32!HrCreateAsyncArgSet+0×479
    206df8e8 650f8221 MSO!Ordinal381+0×48d
    206df908 650f80a9 MSO!Ordinal9712+0×237
    206df924 650f32d6 MSO!Ordinal9712+0xbf
    206df958 650efe05 MSO!Ordinal5368+0×382
    206df9b4 7725338a MSO!MsoFInitOffice+0×363
    206df9c0 77789f72 kernel32!BaseThreadInitThunk+0xe
    206dfa00 77789f45 ntdll!__RtlUserThreadStart+0×70
    206dfa18 00000000 ntdll!_RtlUserThreadStart+0×1b

    Because we have the stack unwind warning we double check the return address to verify that OUTLMIME module called heap free function. The call involves triple indirection of 58149f04 pointer address:


    0:048> ub 58110076
    OUTLMIME!MimeOleInetDateToFileTime+0xd550:
    58110064 8b0f mov ecx,dword ptr [edi]
    58110066 3bcb cmp ecx,ebx
    58110068 740e je OUTLMIME!MimeOleInetDateToFileTime+0xd564 (58110078)
    5811006a a1049f1458 mov eax,dword ptr [OUTLMIME!HrGetMIMEStreamForMAPIMsg+0xe528 (58149f04)]
    5811006f 8b10 mov edx,dword ptr [eax]
    58110071 51 push ecx
    58110072 50 push eax
    58110073 ff5214 call dword ptr [edx+14h]

    0:048> dps poi(poi(58149f04))+14 L1
    76b97264 76aa6e4e ole32!CRetailMalloc_Free [d:\w7rtm\com\ole32\com\class\memapi.cxx @ 680]

    0:048> !heap -s
    **************************************************************
    * *
    * HEAP ERROR DETECTED *
    * *
    **************************************************************

    Details:

    Heap address: 00280000
    Error address: 1cecd3e8
    Error type: HEAP_FAILURE_BLOCK_NOT_BUSY
    Details: The caller performed an operation (such as a free
    or a size check) that is illegal on a free block.
    Follow-up: Check the error’s stack trace to find the culprit.

    Stack trace:
    777ce045: ntdll!RtlFreeHeap+0×00000064
    76aa6e6a: ole32!CRetailMalloc_Free+0×0000001c
    58110076: OUTLMIME!MimeOleInetDateToFileTime+0×0000d562
    581100e9: OUTLMIME!MimeOleInetDateToFileTime+0×0000d5d5
    5811051d: OUTLMIME!MimeOleInetDateToFileTime+0×0000da09
    771562fa: user32!InternalCallWinProc+0×00000023
    77156d3a: user32!UserCallWinProcCheckWow+0×00000109
    771577c4: user32!DispatchMessageWorker+0×000003bc
    77157bca: user32!DispatchMessageA+0×0000000f
    581d74e6: OUTLPH!DllGetClassObject+0×00005616
    581e04a3: OUTLPH!DllGetClassObject+0×0000e5d3
    581df9ac: OUTLPH!DllGetClassObject+0×0000dadc
    6e558488: OLMAPI32!HrCreateAsyncArgSet+0×00000479
    650fa17d: MSO!Ordinal381+0×0000048d
    650f8221: MSO!Ordinal9712+0×00000237
    650f80a9: MSO!Ordinal9712+0×000000bf
    […]

    0:048> !heap -x 1cecd3e8
    Entry User Heap Segment Size PrevSize Unused Flags
    —————————————————————————–
    1cecd3e8 1cecd3f0 00280000 0f945f18 20 - 0 LFH;free

  6. Dmitry Vostokov Says:

    0:000> !heap -s -v

    Details:

    Heap address: 00cb0000
    Error address: 103f4550
    Error type: HEAP_FAILURE_BLOCK_NOT_BUSY
    Details: The caller performed an operation (such as a free
    or a size check) that is illegal on a free block.
    Follow-up: Check the error’s stack trace to find the culprit.

    Stack trace:
    772ec3bd: ntdll!RtlpFreeHeapInternal+0×000000db
    772ac6dc: ntdll!RtlFreeHeap+0×0000002c
    747a8eab: combase!CRetailMalloc_Free+0×0000001b
    6a3537b1: hlink!CMalloc::Free+0×00000031
    6a3536b3: hlink!HLNK_Unk::Release+0×000000f3
    2e5933a8: PPCORE!PPMain+0×00406423
    2e2847ba: PPCORE!PPMain+0×000f7835
    2e26a9ec: PPCORE!PPMain+0×000dda67
    2e26a8dd: PPCORE!PPMain+0×000dd958
    2e26a76e: PPCORE!PPMain+0×000dd7e9
    2e26a6e7: PPCORE!PPMain+0×000dd762
    2e26a68c: PPCORE!PPMain+0×000dd707
    2e26a616: PPCORE!PPMain+0×000dd691
    2e26a5ec: PPCORE!PPMain+0×000dd667
    54a6647e: OART!Ordinal834+0×00000057
    54ade32c: OART!Ordinal971+0×00000094

    0:000> !heap -x 103f4550
    Entry User Heap Segment Size PrevSize Unused Flags
    —————————————————————————–
    103f4550 103f4558 00cb0000 1010ffa8 e0 - 0 LFH;free

Leave a Reply