« Crash Dump Analysis Patterns (Part 1)
Crash Dump Analysis Patterns (Part 3) »

Crash Dump Analysis Patterns (Part 2)

Another pattern I would like to discuss is Dynamic Memory Corruption (and its user and kernel variants called Heap Corruption and Pool Corruption). You might have already guessed it :-) It is so ubiquitous. And its manifestations are random and usually crashes happen far away from the original corruption point. In your user mode and space part of exception threads (don’t forget about Multiple Exceptions pattern) you would see something like this:

ntdll!RtlpCoalesceFreeBlocks+0x10c
ntdll!RtlFreeHeap+0x142
MSVCRT!free+0xda
componentA!xxx

or this

ntdll!RtlpCoalesceFreeBlocks+0x10c
ntdll!RtlpExtendHeap+0x1c1
ntdll!RtlAllocateHeap+0x3b6
componentA!xxx

or any similar variants and you need to know exact component that corrupted application heap (which usually is not the same as componentA.dll you see in crashed thread stack).

For this common recurrent problem we have a general solution: enable heap checking. This general solution has many variants applied in a specific context:

  • parameter value checking for heap functions

  • user space software heap checks before or after certain checkpoints (like “malloc”/”new” and/or “free”/”delete” calls): usually implemented by checking various fill patterns, etc.

  • hardware/OS supported heap checks (like using guard and nonaccessible pages to trap buffer overruns)

The latter variant is the mostly used according to my experience and mainly due to the fact that most heap corruptions originate from buffer overflows. And it is easier to rely on instant MMU support than on checking fill patterns. Here is the article from Citrix support web site describing how you can enable full page heap. It uses specific process as an example: Citrix Independent Management Architecture (IMA) service but you can substitute any application name you are interested in debugging:

How to enable full page heap

and another article:

How to check in a user dump that full page heap was enabled

The following Microsoft article discusses various heap related checks:

How to use Pageheap.exe in Windows XP and Windows 2000

The Windows kernel analog to user mode and space heap corruption is called page and nonpaged pool corruption. If we consider Windows kernel pools as variants of heap then exactly the same techniques are applicable there, for example, the so called special pool enabled by Driver Verifier is implemented by nonaccessible pages. Refer to the following Microsoft article for further details:

How to use the special pool feature to isolate pool damage

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Tuesday, October 31st, 2006 at 4:38 pm and is filed under Crash Dump Analysis, Crash Dump Patterns, Kernel Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

20 Responses to “Crash Dump Analysis Patterns (Part 2)”

  1. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 23a) Says:
    August 19th, 2007 at 7:57 pm

    Double Free pattern:

    […] bugs lead to Dynamic Memory Corruption. The reason why Double Free deserves its own pattern name is the fact that either debug runtime […]

  2. Management Bits and Tips » Blog Archive » Project Failure Analysis Patterns (Part 2) Says:
    January 7th, 2008 at 2:44 pm

    […] are added to a pile or removed from it. Therefore we have just established the mapping between Dynamic Memory Corruption pattern from crash dump analysis domain to Project Artifact Corruption […]

  3. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 2b) Says:
    March 13th, 2008 at 1:00 pm

    […] is an additional kernel example to my old Dynamic Memory Corruption pattern. If kernel pools are corrupt then calls that allocate or free memory result in bugchecks […]

  4. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 71) Says:
    July 11th, 2008 at 8:56 pm

    […] memory corruption patterns in user and kernel spaces are specializations of one big parent pattern called Corrupt Structure […]

  5. !analyze -v : 크래쉬 덤프 분석 패턴 (Part 2) Says:
    September 17th, 2008 at 9:45 am

    […] http://www.dumpanalysis.org/blog/index.php/2006/10/31/crash-dump-analysis-patterns-part-2/ […]

  6. Crash Dump Analysis » Blog Archive » Exception and deadlock: pattern cooperation Says:
    September 18th, 2008 at 3:08 pm

    […] was an exception while loading a DLL. Applying exception context WinDbg command .cxr reveals heap corruption […]

  7. Crash Dump Analysis » Blog Archive » Heap Corruption Says:
    November 21st, 2008 at 4:39 pm

    […] Dynamic Memory Corruption (process heap) […]

  8. Crash Dump Analysis » Blog Archive » Dynamic Memory Corruption Patterns Says:
    February 17th, 2009 at 9:44 am

    […] Dynamic Memory Corruption (process heap) […]

  9. Crash Dump Analysis » Blog Archive » Manual dump, dynamic memory corruption, blocked threads, stack trace collection, multiple exceptions, wait chains and deadlock: pattern cooperation Says:
    March 8th, 2009 at 1:17 am

    […] see the exeption was dispatched because of heap corruption and the unhandled exception filter is blocked waiting for a critical section. We can immediately […]

  10. Crash Dump Analysis » Blog Archive » Manual dump, wait chain, blocked thread, dynamic memory corruption and historical information: pattern cooperation Says:
    March 14th, 2009 at 8:38 am

    […] exception filter raising a hard error message box. Applying the new exception context we confirm heap corruption […]

  11. La billeterie » Blog Archive » Analyse mémoire sous windows Says:
    April 22nd, 2009 at 9:56 am

    […] des options de gflags.exe Tous les flags de gflags Gflags : Enable page heap Full/Standard enable heap checking La structure des blocs mémoire quand quand le page heap est activé appverifier The Structure of a […]

  12. Crash Dump Analysis » Blog Archive » Heap corruption, module variety, execution residue, coincidental symbolic information and critical section corruption: pattern cooperation Says:
    May 1st, 2009 at 2:54 pm

    […] environments or insufficiently tested in multi-threaded environments. Many such crashes result from dynamic memory corruption of a process […]

  13. Crash Dump Analysis » Blog Archive » Crash Dump Analysis AntiPatterns (Part 13) Says:
    July 9th, 2009 at 3:54 pm

    […] without questioning them or not considering possible exceptions. For example, the usual advise to heap corruption signs in process memory dumps is to ask to enable full page heap. However page heap helps to […]

  14. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 3) Says:
    March 10th, 2010 at 3:39 pm

    […] Today we introduce an icon for Dynamic Memory Corruption (process heap) pattern: […]

  15. Crash Dump Analysis » Blog Archive » Strong process coupling, stack trace collection, critical section coruption and wait chains, message box, self-diagnosis and hidden exception and dynamic memory corruption: pattern cooperation Says:
    April 26th, 2010 at 7:54 pm

    […] It finally looks like a heap corruption: […]

  16. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 110) Says:
    October 18th, 2010 at 2:14 pm

    […] Experts Magazine Online Shared Buffer Overwrite differs from Local Buffer Overflow and heap / pool memory corruption patterns in not writing over control structures situated at dynamically […]

  17. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 117) Says:
    November 29th, 2010 at 11:48 pm

    […] to functions. Here we look at invalid heap block parameter specialization. It is different from heap corruption or double free pattern because no corruption happens in heap structures before detection and the […]

  18. Dmitry Vostokov Says:
    February 12th, 2012 at 11:17 pm

    !heap -s -v verifies heap blocks:

    0:001> !heap -s -v
    **************************************************************
    * *
    * HEAP ERROR DETECTED *
    * *
    **************************************************************

    Details:

    Error address: 00740f28
    Heap handle: 00740000
    Error type heap_failure_multiple_entries_corruption (4)
    Last known valid blocks: before - 007409e8, after - 007416b8
    Stack trace:
    77b6fc76: ntdll!RtlpAnalyzeHeapFailure+0x0000025b
    77b29ef1: ntdll!RtlpCoalesceFreeBlocks+0x00000060
    77ad2d07: ntdll!RtlpFreeHeap+0x000001f4
    77ad2bf2: ntdll!RtlFreeHeap+0x00000142
    752914d1: kernel32!HeapFree+0x00000014
    010b11f0: Application+0x000011f0
    010b1274: Application+0x00001274
    010b1310: Application+0x00001310
    75293677: kernel32!BaseThreadInitThunk+0x0000000e
    77ad9f02: ntdll!__RtlUserThreadStart+0x00000070
    77ad9ed5: ntdll!_RtlUserThreadStart+0x0000001b
    LFH Key : 0x7c150f40
    Termination on corruption : DISABLED
    Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
    (k) (k) (k) (k) length blocks cont. heap
    -----------------------------------------------------------------------------
    .004c0000 00000002 1024 104 104 2 1 1 0 0 LFH
    .ERROR: Block 007416b8 previous size f2 does not match previous block size 44
    HEAP 00740000 (Seg 00740000) At 007416b8 Error: invalid block Previous

    00740000 00001002 64 12 64 3 2 1 0 0
    -----------------------------------------------------------------------------

  19. Dmitry Vostokov Says:
    May 21st, 2013 at 8:07 pm

    Sometimes we may have a buffer underflow and full page heap which places allocations at the end of pages will not catch the moment of corruption. Here we need to use backwards full page heap:

    gflags /p /enable ImageFile /full /backwards

    Debugging TV frames episode 0×26 has full such example:
    http://www.debugging.tv

  20. Dmitry Vostokov Says:
    July 11th, 2013 at 6:00 pm

    The new WinDbg 6.2.9200.20512 !analyze -v detects invalid heap calls in case “heap termination on corruption” is not enabled (by default on legacy 32-bit apps). In the past it was possible to see that only with heap verification command such as !heap -s -v or via dps ntdll!RtlpHeapFailureInfo. PS. Visual C++ 2012 enables heap termination on corruption by default even for 32-bit targets according to SDL guidelines: http://msdn.microsoft.com/en-us/library/windows/desktop/cc307399.aspx

    0:001> !heap -s -v
    […]
    Heap address: 00580000
    Error address: 005c1a2a
    Error type: HEAP_FAILURE_INVALID_ARGUMENT
    Details: The caller tried to a free a block at an invalid
    (unaligned) address.
    Follow-up: Check the error’s stack trace to find the culprit.

    Stack trace:
    7799dff5: ntdll!RtlFreeHeap+0×00000064
    767514dd: kernel32!HeapFree+0×00000014
    0138140f: AppD8!free+0×0000001a
    0138134d: AppD8!StartModeling+0×0000001d
    0138121a: AppD8!WndProc+0×0000007a
    76f162fa: USER32!InternalCallWinProc+0×00000023
    76f16d3a: USER32!UserCallWinProcCheckWow+0×00000109
    76f177c4: USER32!DispatchMessageWorker+0×000003bc
    76f1788a: USER32!DispatchMessageW+0×0000000f
    0138109d: AppD8!wWinMain+0×0000009d
    0138152a: AppD8!__tmainCRTStartup+0×000000fd
    767533aa: kernel32!BaseThreadInitThunk+0×0000000e
    77959ef2: ntdll!__RtlUserThreadStart+0×00000070
    77959ec5: ntdll!_RtlUserThreadStart+0×0000001b

    LFH Key : 0×3d43a3cb
    Termination on corruption : DISABLED

    0:001> !analyze -v
    […]
    BUGCHECK_STR: APPLICATION_FAULT_ACTIONABLE_HEAP_CORRUPTION_heap_failure_invalid_argument
    […]
    STACK_TEXT:
    77a242a0 7799dff5 ntdll!RtlFreeHeap+0×64
    77a242a4 767514dd kernel32!HeapFree+0×14
    77a242a8 0138140f appd8!free+0×1a
    77a242ac 0138134d appd8!StartModeling+0×1d
    77a242b0 0138121a appd8!WndProc+0×7a
    77a242b4 76f162fa user32!InternalCallWinProc+0×23
    77a242b8 76f16d3a user32!UserCallWinProcCheckWow+0×109
    77a242bc 76f177c4 user32!DispatchMessageWorker+0×3bc
    77a242c0 76f1788a user32!DispatchMessageW+0xf
    77a242c4 0138109d appd8!wWinMain+0×9d
    77a242c8 0138152a appd8!__tmainCRTStartup+0xfd
    77a242cc 767533aa kernel32!BaseThreadInitThunk+0xe
    77a242d0 77959ef2 ntdll!__RtlUserThreadStart+0×70
    77a242d4 77959ec5 ntdll!_RtlUserThreadStart+0×1b
    […]

    0:001> dps ntdll!RtlpHeapFailureInfo
    77a24268 00000000
    77a2426c 00000000
    77a24270 00000009
    77a24274 00580000
    77a24278 005c1a2a
    77a2427c 00000000
    77a24280 00000000
    77a24284 00000000
    77a24288 00000000
    77a2428c 00000000
    77a24290 00000000
    77a24294 00000000
    77a24298 00000000
    77a2429c 00000000
    77a242a0 7799dff5 ntdll!RtlFreeHeap+0×64
    77a242a4 767514dd kernel32!HeapFree+0×14
    77a242a8 0138140f AppD8!free+0×1a
    77a242ac 0138134d AppD8!StartModeling+0×1d
    77a242b0 0138121a AppD8!WndProc+0×7a
    77a242b4 76f162fa USER32!InternalCallWinProc+0×23
    77a242b8 76f16d3a USER32!UserCallWinProcCheckWow+0×109
    77a242bc 76f177c4 USER32!DispatchMessageWorker+0×3bc
    77a242c0 76f1788a USER32!DispatchMessageW+0xf
    77a242c4 0138109d AppD8!wWinMain+0×9d
    77a242c8 0138152a AppD8!__tmainCRTStartup+0xfd
    77a242cc 767533aa kernel32!BaseThreadInitThunk+0xe
    77a242d0 77959ef2 ntdll!__RtlUserThreadStart+0×70
    77a242d4 77959ec5 ntdll!_RtlUserThreadStart+0×1b
    77a242d8 00000000
    77a242dc 00000000
    77a242e0 00000000
    77a242e4 00000000

    0:001> ~*k

    0 Id: 1d74.fd4 Suspend: 1 Teb: 7efdd000 Unfrozen
    ChildEBP RetAddr
    001cfa3c 76f1790d USER32!NtUserGetMessage+0×15
    001cfa58 0138106f USER32!GetMessageW+0×33
    001cfa90 0138152a AppD8!wWinMain+0×6f
    001cfadc 767533aa AppD8!__tmainCRTStartup+0xfd
    001cfae8 77959ef2 kernel32!BaseThreadInitThunk+0xe
    001cfb28 77959ec5 ntdll!__RtlUserThreadStart+0×70
    001cfb40 00000000 ntdll!_RtlUserThreadStart+0×1b

    # 1 Id: 1d74.e98 Suspend: 1 Teb: 7efda000 Unfrozen
    ChildEBP RetAddr
    0118fbb4 779bf896 ntdll!DbgBreakPoint
    0118fbe4 767533aa ntdll!DbgUiRemoteBreakin+0×3c
    0118fbf0 77959ef2 kernel32!BaseThreadInitThunk+0xe
    0118fc30 77959ec5 ntdll!__RtlUserThreadStart+0×70
    0118fc48 00000000 ntdll!_RtlUserThreadStart+0×1b

Leave a Reply

You must be logged in to post a comment.