Crash Dump Analysis Patterns (Part 2)
Another pattern I would like to discuss is Dynamic Memory Corruption (and its user and kernel variants called Heap Corruption and Pool Corruption). You might have already guessed it 
It is so ubiquitous. And its manifestations are random and usually crashes happen far away from the original corruption point. In your user mode and space part of exception threads (don’t forget about Multiple Exceptions pattern) you would see something like this:
ntdll!RtlpCoalesceFreeBlocks+0x10c
ntdll!RtlFreeHeap+0x142
MSVCRT!free+0xda
componentA!xxx
or this
ntdll!RtlpCoalesceFreeBlocks+0x10c
ntdll!RtlpExtendHeap+0x1c1
ntdll!RtlAllocateHeap+0x3b6
componentA!xxx
or any similar variants and you need to know exact component that corrupted application heap (which usually is not the same as componentA.dll you see in crashed thread stack).
For this common recurrent problem we have a general solution: enable heap checking. This general solution has many variants applied in a specific context:
-
parameter value checking for heap functions
-
user space software heap checks before or after certain checkpoints (like “malloc”/”new” and/or “free”/”delete” calls): usually implemented by checking various fill patterns, etc.
-
hardware/OS supported heap checks (like using guard and nonaccessible pages to trap buffer overruns)
The latter variant is the mostly used according to my experience and mainly due to the fact that most heap corruptions originate from buffer overflows. And it is easier to rely on instant MMU support than on checking fill patterns. Here is the article from Citrix support web site describing how you can enable full page heap. It uses specific process as an example: Citrix Independent Management Architecture (IMA) service but you can substitute any application name you are interested in debugging:
and another article:
How to check in a user dump that full page heap was enabled
The following Microsoft article discusses various heap related checks:
How to use Pageheap.exe in Windows XP and Windows 2000
The Windows kernel analog to user mode and space heap corruption is called page and nonpaged pool corruption. If we consider Windows kernel pools as variants of heap then exactly the same techniques are applicable there, for example, the so called special pool enabled by Driver Verifier is implemented by nonaccessible pages. Refer to the following Microsoft article for further details:
How to use the special pool feature to isolate pool damage
- Dmitry Vostokov @ DumpAnalysis.org -
August 19th, 2007 at 7:57 pm
Double Free pattern:
[…] bugs lead to Dynamic Memory Corruption. The reason why Double Free deserves its own pattern name is the fact that either debug runtime […]
January 7th, 2008 at 2:44 pm
[…] are added to a pile or removed from it. Therefore we have just established the mapping between Dynamic Memory Corruption pattern from crash dump analysis domain to Project Artifact Corruption […]
March 13th, 2008 at 1:00 pm
[…] is an additional kernel example to my old Dynamic Memory Corruption pattern. If kernel pools are corrupt then calls that allocate or free memory result in bugchecks […]
July 11th, 2008 at 8:56 pm
[…] memory corruption patterns in user and kernel spaces are specializations of one big parent pattern called Corrupt Structure […]
September 17th, 2008 at 9:45 am
[…] http://www.dumpanalysis.org/blog/index.php/2006/10/31/crash-dump-analysis-patterns-part-2/ […]
September 18th, 2008 at 3:08 pm
[…] was an exception while loading a DLL. Applying exception context WinDbg command .cxr reveals heap corruption […]
November 21st, 2008 at 4:39 pm
[…] Dynamic Memory Corruption (process heap) […]
February 17th, 2009 at 9:44 am
[…] Dynamic Memory Corruption (process heap) […]
March 8th, 2009 at 1:17 am
[…] see the exeption was dispatched because of heap corruption and the unhandled exception filter is blocked waiting for a critical section. We can immediately […]
March 14th, 2009 at 8:38 am
[…] exception filter raising a hard error message box. Applying the new exception context we confirm heap corruption […]
April 22nd, 2009 at 9:56 am
[…] des options de gflags.exe Tous les flags de gflags Gflags : Enable page heap Full/Standard enable heap checking La structure des blocs mémoire quand quand le page heap est activé appverifier The Structure of a […]
May 1st, 2009 at 2:54 pm
[…] environments or insufficiently tested in multi-threaded environments. Many such crashes result from dynamic memory corruption of a process […]
July 9th, 2009 at 3:54 pm
[…] without questioning them or not considering possible exceptions. For example, the usual advise to heap corruption signs in process memory dumps is to ask to enable full page heap. However page heap helps to […]
March 10th, 2010 at 3:39 pm
[…] Today we introduce an icon for Dynamic Memory Corruption (process heap) pattern: […]
April 26th, 2010 at 7:54 pm
[…] It finally looks like a heap corruption: […]
October 18th, 2010 at 2:14 pm
[…] Experts Magazine Online Shared Buffer Overwrite differs from Local Buffer Overflow and heap / pool memory corruption patterns in not writing over control structures situated at dynamically […]
November 29th, 2010 at 11:48 pm
[…] to functions. Here we look at invalid heap block parameter specialization. It is different from heap corruption or double free pattern because no corruption happens in heap structures before detection and the […]
February 12th, 2012 at 11:17 pm
!heap -s -v verifies heap blocks:
0:001> !heap -s -v**************************************************************
* *
* HEAP ERROR DETECTED *
* *
**************************************************************
Details:Error address: 00740f28Heap handle: 00740000
Error type heap_failure_multiple_entries_corruption (4)
Last known valid blocks: before - 007409e8, after - 007416b8
Stack trace:
77b6fc76: ntdll!RtlpAnalyzeHeapFailure+0x0000025b
77b29ef1: ntdll!RtlpCoalesceFreeBlocks+0x00000060
77ad2d07: ntdll!RtlpFreeHeap+0x000001f4
77ad2bf2: ntdll!RtlFreeHeap+0x00000142
752914d1: kernel32!HeapFree+0x00000014
010b11f0: Application+0x000011f0
010b1274: Application+0x00001274
010b1310: Application+0x00001310
75293677: kernel32!BaseThreadInitThunk+0x0000000e
77ad9f02: ntdll!__RtlUserThreadStart+0x00000070
77ad9ed5: ntdll!_RtlUserThreadStart+0x0000001b
LFH Key : 0x7c150f40
Termination on corruption : DISABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
.004c0000 00000002 1024 104 104 2 1 1 0 0 LFH
.ERROR: Block 007416b8 previous size f2 does not match previous block size 44
HEAP 00740000 (Seg 00740000) At 007416b8 Error: invalid block Previous
00740000 00001002 64 12 64 3 2 1 0 0-----------------------------------------------------------------------------
May 21st, 2013 at 8:07 pm
Sometimes we may have a buffer underflow and full page heap which places allocations at the end of pages will not catch the moment of corruption. Here we need to use backwards full page heap:
gflags /p /enable ImageFile /full /backwards
Debugging TV frames episode 0×26 has full such example:
http://www.debugging.tv
July 11th, 2013 at 6:00 pm
The new WinDbg 6.2.9200.20512 !analyze -v detects invalid heap calls in case “heap termination on corruption” is not enabled (by default on legacy 32-bit apps). In the past it was possible to see that only with heap verification command such as !heap -s -v or via dps ntdll!RtlpHeapFailureInfo. PS. Visual C++ 2012 enables heap termination on corruption by default even for 32-bit targets according to SDL guidelines: http://msdn.microsoft.com/en-us/library/windows/desktop/cc307399.aspx
0:001> !heap -s -v
[…]
Heap address: 00580000
Error address: 005c1a2a
Error type: HEAP_FAILURE_INVALID_ARGUMENT
Details: The caller tried to a free a block at an invalid
(unaligned) address.
Follow-up: Check the error’s stack trace to find the culprit.
Stack trace:
7799dff5: ntdll!RtlFreeHeap+0×00000064
767514dd: kernel32!HeapFree+0×00000014
0138140f: AppD8!free+0×0000001a
0138134d: AppD8!StartModeling+0×0000001d
0138121a: AppD8!WndProc+0×0000007a
76f162fa: USER32!InternalCallWinProc+0×00000023
76f16d3a: USER32!UserCallWinProcCheckWow+0×00000109
76f177c4: USER32!DispatchMessageWorker+0×000003bc
76f1788a: USER32!DispatchMessageW+0×0000000f
0138109d: AppD8!wWinMain+0×0000009d
0138152a: AppD8!__tmainCRTStartup+0×000000fd
767533aa: kernel32!BaseThreadInitThunk+0×0000000e
77959ef2: ntdll!__RtlUserThreadStart+0×00000070
77959ec5: ntdll!_RtlUserThreadStart+0×0000001b
LFH Key : 0×3d43a3cb
Termination on corruption : DISABLED
0:001> !analyze -v
[…]
BUGCHECK_STR: APPLICATION_FAULT_ACTIONABLE_HEAP_CORRUPTION_heap_failure_invalid_argument
[…]
STACK_TEXT:
77a242a0 7799dff5 ntdll!RtlFreeHeap+0×64
77a242a4 767514dd kernel32!HeapFree+0×14
77a242a8 0138140f appd8!free+0×1a
77a242ac 0138134d appd8!StartModeling+0×1d
77a242b0 0138121a appd8!WndProc+0×7a
77a242b4 76f162fa user32!InternalCallWinProc+0×23
77a242b8 76f16d3a user32!UserCallWinProcCheckWow+0×109
77a242bc 76f177c4 user32!DispatchMessageWorker+0×3bc
77a242c0 76f1788a user32!DispatchMessageW+0xf
77a242c4 0138109d appd8!wWinMain+0×9d
77a242c8 0138152a appd8!__tmainCRTStartup+0xfd
77a242cc 767533aa kernel32!BaseThreadInitThunk+0xe
77a242d0 77959ef2 ntdll!__RtlUserThreadStart+0×70
77a242d4 77959ec5 ntdll!_RtlUserThreadStart+0×1b
[…]
0:001> dps ntdll!RtlpHeapFailureInfo
77a24268 00000000
77a2426c 00000000
77a24270 00000009
77a24274 00580000
77a24278 005c1a2a
77a2427c 00000000
77a24280 00000000
77a24284 00000000
77a24288 00000000
77a2428c 00000000
77a24290 00000000
77a24294 00000000
77a24298 00000000
77a2429c 00000000
77a242a0 7799dff5 ntdll!RtlFreeHeap+0×64
77a242a4 767514dd kernel32!HeapFree+0×14
77a242a8 0138140f AppD8!free+0×1a
77a242ac 0138134d AppD8!StartModeling+0×1d
77a242b0 0138121a AppD8!WndProc+0×7a
77a242b4 76f162fa USER32!InternalCallWinProc+0×23
77a242b8 76f16d3a USER32!UserCallWinProcCheckWow+0×109
77a242bc 76f177c4 USER32!DispatchMessageWorker+0×3bc
77a242c0 76f1788a USER32!DispatchMessageW+0xf
77a242c4 0138109d AppD8!wWinMain+0×9d
77a242c8 0138152a AppD8!__tmainCRTStartup+0xfd
77a242cc 767533aa kernel32!BaseThreadInitThunk+0xe
77a242d0 77959ef2 ntdll!__RtlUserThreadStart+0×70
77a242d4 77959ec5 ntdll!_RtlUserThreadStart+0×1b
77a242d8 00000000
77a242dc 00000000
77a242e0 00000000
77a242e4 00000000
0:001> ~*k
0 Id: 1d74.fd4 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
001cfa3c 76f1790d USER32!NtUserGetMessage+0×15
001cfa58 0138106f USER32!GetMessageW+0×33
001cfa90 0138152a AppD8!wWinMain+0×6f
001cfadc 767533aa AppD8!__tmainCRTStartup+0xfd
001cfae8 77959ef2 kernel32!BaseThreadInitThunk+0xe
001cfb28 77959ec5 ntdll!__RtlUserThreadStart+0×70
001cfb40 00000000 ntdll!_RtlUserThreadStart+0×1b
# 1 Id: 1d74.e98 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
0118fbb4 779bf896 ntdll!DbgBreakPoint
0118fbe4 767533aa ntdll!DbgUiRemoteBreakin+0×3c
0118fbf0 77959ef2 kernel32!BaseThreadInitThunk+0xe
0118fc30 77959ec5 ntdll!__RtlUserThreadStart+0×70
0118fc48 00000000 ntdll!_RtlUserThreadStart+0×1b