Crash Dump Analysis Patterns (Part 13h)
Monday, April 9th, 2012Allocated dynamic memory such as process heap can remain reserved after deallocation and its virtual memory region might become unavailable for usage. One example of this I encountered recently while debugging a .NET service. During a peak usage it reported various out-of-memory events but its managed heap was healthy and didn’t consume much. However, its process heap statistics showed a large reserved heap segment missing in a similar memory dump from a development environment. Remaining allocated entries in that heap segment contained a specific module hint that allowed us to suggest removing a 3rd-party product from a production environment.
In order to provide the proof of that possible scenario of reserved heap regions we created a special modeling application:
int _tmain(int argc, _TCHAR* argv[])
{
static char *pAlloc[1000000];
for (int i = 0; i < 1000000; i++)
{
pAlloc[i] = (char *)malloc (1000);
}
getc(stdin);
for (int i = 0; i < 1000000; i++)
{
free(pAlloc[i]);
}
getc(stdin);
return 0;
}
Here’s the debugging log:
0:001> .symfix c:\mss
0:001> .reload
Reloading current modules
.....
After allocation:
0:001> !heap -s
LFH Key : 0x156356e0
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
—————————————————————————–
00520000 00000002 1024 112 1024 8 1 1 0 0 LFH
007e0000 00001002 1019328 1012444 1019328 131 68 67 0 0 LFH
—————————————————————————–
0:001> g
(1588.14b0): Break instruction exception - code 80000003 (first chance)
eax=7efda000 ebx=00000000 ecx=00000000 edx=770ff85a esi=00000000 edi=00000000
eip=7707000c esp=00f0f7e4 ebp=00f0f810 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!DbgBreakPoint:
7707000c cc int 3
After deallocation:
0:001> !heap -s
LFH Key : 0x156356e0
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
—————————————————————————–
00520000 00000002 1024 112 1024 8 1 1 0 0 LFH
007e0000 00001002 1019328 73040 1019328 71365 419 165 0 0 LFH
External fragmentation 97 % (419 free blocks)
Virtual address fragmentation 92 % (165 uncommited ranges)
—————————————————————————–
0:001> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 26 3fbe7000 (1019.902 Mb) 49.80%
<unclassified> 752 3f8ec000 (1016.922 Mb) 98.92% 49.66%
Image 41 76b000 ( 7.418 Mb) 0.72% 0.36%
Stack 6 200000 ( 2.000 Mb) 0.19% 0.10%
MemoryMappedFile 8 1af000 ( 1.684 Mb) 0.16% 0.08%
TEB 2 2000 ( 8.000 kb) 0.00% 0.00%
PEB 1 1000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 734 3f8a2000 (1016.633 Mb) 98.89% 49.64%
MEM_IMAGE 68 9b8000 ( 9.719 Mb) 0.95% 0.47%
MEM_MAPPED 8 1af000 ( 1.684 Mb) 0.16% 0.08%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 26 3fbe7000 (1019.902 Mb) 49.80%
MEM_RESERVE 374 3f6e8000 (1014.906 Mb) 98.72% 49.56%
MEM_COMMIT 436 d21000 ( 13.129 Mb) 1.28% 0.64%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 383 725000 ( 7.145 Mb) 0.69% 0.35%
PAGE_EXECUTE_READ 10 414000 ( 4.078 Mb) 0.40% 0.20%
PAGE_READONLY 29 1cd000 ( 1.801 Mb) 0.18% 0.09%
PAGE_WRITECOPY 10 12000 ( 72.000 kb) 0.01% 0.00%
PAGE_READWRITE|PAGE_GUARD 4 9000 ( 36.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 3f0c0000 33050000 ( 816.313 Mb)
<unclassified> 158a1000 fcf000 ( 15.809 Mb)
Image 1083000 3d1000 ( 3.816 Mb)
Stack 200000 fd000 (1012.000 kb)
MemoryMappedFile 7efe5000 fb000 (1004.000 kb)
TEB 7efda000 1000 ( 4.000 kb)
PEB 7efde000 1000 ( 4.000 kb)
We see that free memory available for allocation is only 816 Mb.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
