Archive for the ‘Pattern Models’ Category
Thursday, October 30th, 2014
The set of memory dumps that prompted to introduce Insufficient Memory pattern for stack trace database also prompted to include a variant of Memory Leak pattern related to regions of virtual memory address space. We created this simple modeling application:
int _tmain(int argc, _TCHAR* argv[])
{
int i,j;
for (i = 1; i < 1000; ++i)
{
for (j = 1; j < 1000; ++j)
{
VirtualAlloc(NULL, 0x10000, MEM_RESERVE,
PAGE_EXECUTE_READWRITE);
}
getc(stdin);
}
return 0;
}
We allocated only reserved memory regions. Committing them would probably at some stage manifest Insufficient Memory patterns for committed memory and physical memory. So we took a few consecutive memory dumps and see the ever increasing number of regions allocated at greater and greater virtual addresses:
0:000> !address
[...]
* 0`04070000 0`04080000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04080000 0`04090000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04090000 0`040a0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`040a0000 0`040b0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`040b0000 0`040c0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`040c0000 0`040d0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`040d0000 0`040e0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`040e0000 0`040f0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`040f0000 0`04100000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04100000 0`04110000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04110000 0`04120000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04120000 0`04130000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04130000 0`04140000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`04140000 0`04260000 0`00120000 MEM_FREE PAGE_NOACCESS Free
[...]
0:000> !address
[...]
* 0`2eec0000 0`2eed0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2eed0000 0`2eee0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2eee0000 0`2eef0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2eef0000 0`2ef00000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef00000 0`2ef10000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef10000 0`2ef20000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef20000 0`2ef30000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef30000 0`2ef40000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef40000 0`2ef50000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef50000 0`2ef60000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef60000 0`2ef70000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef70000 0`2ef80000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef80000 0`2ef90000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2ef90000 0`2efa0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2efa0000 0`2efb0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2efb0000 0`2efc0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2efc0000 0`2efd0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2efd0000 0`2efe0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2efe0000 0`2eff0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2eff0000 0`2f000000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2f000000 0`2f010000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`2f010000 0`2f170000 0`00160000 MEM_FREE PAGE_NOACCESS Free
[...]
0:000> !address
[...]
* 0`697f0000 0`69800000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69800000 0`69810000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69810000 0`69820000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69820000 0`69830000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69830000 0`69840000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69840000 0`69850000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69850000 0`69860000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69860000 0`69870000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69870000 0`69880000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69880000 0`69890000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`69890000 0`698a0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`698a0000 0`699e0000 0`00140000 MEM_FREE PAGE_NOACCESS Free
[...]
0:000> !address
[...]
* 0`c08c0000 0`c08d0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c08d0000 0`c08e0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c08e0000 0`c08f0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c08f0000 0`c0900000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c0900000 0`c0910000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c0910000 0`c0920000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c0920000 0`c0930000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 0`c0930000 0`c0960000 0`00030000 MEM_FREE PAGE_NOACCESS Free
[...]
0:000> !address
[...]
* 1`3d6a0000 1`3d6b0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d6b0000 1`3d6c0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d6c0000 1`3d6d0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d6d0000 1`3d6e0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d6e0000 1`3d6f0000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d6f0000 1`3d700000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d700000 1`3d710000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d710000 1`3d720000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d720000 1`3d730000 0`00010000 MEM_PRIVATE MEM_RESERVE <unclassified>
* 1`3d730000 1`3d7a0000 0`00070000 MEM_FREE PAGE_NOACCESS Free
[...]
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ———– Total Size ——– %ofBusy %ofTotal
Free 15 7fe`c275e000 ( 7.995 Tb) 99.94%
<unclassified> 80928 1`3d193000 ( 4.955 Gb) 99.86% 0.06%
Image 28 0`0034b000 ( 3.293 Mb) 0.06% 0.00%
Stack 6 0`00200000 ( 2.000 Mb) 0.04% 0.00%
MemoryMappedFile 8 0`001af000 ( 1.684 Mb) 0.03% 0.00%
TEB 2 0`00004000 ( 16.000 kb) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 80936 1`3d397000 ( 4.957 Gb) 99.90% 0.06%
MEM_IMAGE 29 0`0034c000 ( 3.297 Mb) 0.06% 0.00%
MEM_MAPPED 8 0`001af000 ( 1.684 Mb) 0.03% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 15 7fe`c275e000 ( 7.995 Tb) 99.94%
MEM_RESERVE 80926 1`3d438000 ( 4.957 Gb) 99.91% 0.06%
MEM_COMMIT 47 0`0045a000 ( 4.352 Mb) 0.09% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ 4 0`001ef000 ( 1.934 Mb) 0.04% 0.00%
PAGE_READONLY 19 0`001de000 ( 1.867 Mb) 0.04% 0.00%
PAGE_READWRITE 17 0`00080000 ( 512.000 kb) 0.01% 0.00%
PAGE_WRITECOPY 5 0`00008000 ( 32.000 kb) 0.00% 0.00%
PAGE_READWRITE|PAGE_GUARD 2 0`00005000 ( 20.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 1`3fac7000 7fd`bdc79000 ( 7.991 Tb)
<unclassified> 0`7f0e0000 0`00f00000 ( 15.000 Mb)
Image 0`77831000 0`00102000 ( 1.008 Mb)
Stack 0`00170000 0`000fb000 (1004.000 kb)
MemoryMappedFile 0`7efe5000 0`000fb000 (1004.000 kb)
TEB 7ff`fffdc000 0`00002000 ( 8.000 kb)
PEB 7ff`fffd3000 0`00001000 ( 4.000 kb)
Examination of such regions for Execution Residue such as Module Hint may point into further troubleshooting directions especially if live debugging is not possible.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Models, Software Defect Construction, x64 Windows | No Comments »
Monday, October 27th, 2014
While working on Insufficient Memory pattern for stack trace database we noticed the expansion of certain memory regions. Of course, after some time expanding region consumes remaining free or reserved space available before some other region. Generalizing from this we may say there can be Insufficient Memory pattern variant for any region expanding region. Region expansion may also be implemented via its move into some over position in memory virtual address space. This movement also has its limits. for example, we created this modeling application and found out it stops reallocating memory long before it reaches 2,000,000,000 byte size:
int _tmain(int argc, _TCHAR* argv[])
{
int i = 100000000;
void *p = malloc(i);
for (i = 200000000; i < 2000000000; i+=100000000)
{
p = realloc(p, i);
getc(stdin);
}
return 0;
}
We took memory dumps after each loop iteration and after 6 or 8 iterations the memory size was constant and there were no further reallocations:
0:000> !heap -s
[...]
Virtual block: 0000000006370000 - 0000000006370000 (size 0000000000000000)
[...]
0:000> !address
[...]
+ 0`00550000 0`06370000 0`05e20000 MEM_FREE PAGE_NOACCESS Free
+ 0`06370000 0`1222d000 0`0bebd000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block]
+ 0`1222d000 0`77710000 0`654e3000 MEM_FREE PAGE_NOACCESS Free
+ 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”]
[…]
0:000> !heap -s
[...]
Virtual block: 0000000012230000 - 0000000012230000 (size 0000000000000000)
[...]
0:000> !address
[...]
+ 0`005d0000 0`12230000 0`11c60000 MEM_FREE PAGE_NOACCESS Free
+ 0`12230000 0`2404b000 0`11e1b000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block]
+ 0`2404b000 0`77710000 0`536c5000 MEM_FREE PAGE_NOACCESS Free
+ 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”]
[…]
0:000> !heap -s
[...]
Virtual block: 0000000024050000 - 0000000024050000 (size 0000000000000000)
[...]
0:000> !address
[...]
+ 0`00590000 0`24050000 0`23ac0000 MEM_FREE PAGE_NOACCESS Free
+ 0`24050000 0`3bdc9000 0`17d79000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block]
+ 0`3bdc9000 0`77710000 0`3b947000 MEM_FREE PAGE_NOACCESS Free
+ 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”]
[…]
We skip a few iterations and finally come to a region that will not move and not increase:
0:000> !heap -s
[...]
Virtual block: 0000000041d30000 - 0000000041d30000 (size 0000000000000000)
[...]
0:000> !address
[...]
+ 0`006c0000 0`41d30000 0`41670000 MEM_FREE PAGE_NOACCESS Free
+ 0`41d30000 0`6b8c3000 0`29b93000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Heap [ID: 0; Handle: 0000000000310000; Type: Large block]
+ 0`6b8c3000 0`77710000 0`0be4d000 MEM_FREE PAGE_NOACCESS Free
+ 0`77710000 0`77711000 0`00001000 MEM_IMAGE MEM_COMMIT PAGE_READONLY Image [kernel32; “C:\windows\system32\kernel32.dll”]
[…]
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Models, Software Defect Construction, x64 Windows | No Comments »
Monday, October 27th, 2014
Once we have seen a sequence of process memory dumps with the largest one almost 4GB. They were all saved from the process with growing memory consumption from 200MB initially. Initially we suspected process heap Memory Leak. However, heap statistics (!heap -s) was normal. There were not even large block allocations. The dumps were also supplied with UMDH logs but their difference only showed Memory Fluctuation and not increase. Stack Trace Collection revealed one Spiking Thread was logging a heap allocation into user mode stack trace database. We could also see that it was Distributed Spike. Inspection of address space showed large number of sequential regions of the same size with Stack Trace Database entries inside. So we concluded that it was stack trace logging Instrumentation Side Effect and advised to limit stack backtrace size in gflags.exe.
To make sure we understood that problem correctly we decided to model it. We didn’t come to the same results probably due to different logging implementation but memory dumps clearly show the possibility of Insufficient Memory pattern variant. Here’s the source code:
void foo20 (int size)
{
free(malloc(size));
}
#define FOO(x,y) void foo##x (int size) { foo##y(size); }
FOO(19,20)
FOO(18,19)
FOO(17,18)
FOO(16,17)
FOO(15,16)
FOO(14,15)
FOO(13,14)
FOO(12,13)
FOO(11,12)
FOO(10,11)
FOO(9,10)
FOO(8,9)
FOO(7,8)
FOO(6,7)
FOO(5,6)
FOO(4,5)
FOO(3,4)
FOO(2,3)
FOO(1,2)
typedef void (*PFN) (int);
#define ARRSZ 20
PFN pfnArr[ARRSZ] = {foo1, foo2, foo3, foo4, foo5, foo6, foo7,
foo8, foo9, foo10, foo11, foo12, foo13, foo14,
foo15, foo16, foo17, foo18, foo19, foo20};
int _tmain(int argc, _TCHAR* argv[])
{
int i;
for (i = 1; i < 1000000000; ++i)
{
pfnArr[i%ARRSZ](i);
}
Sleep(-1);
return 0;
}
It allocates and then freezes heap entries of different size from 1 byte to 1,000,000,000 bytes all with different 20 possible stack traces. We choose different stack traces to increase the number of different {size, stack backtrace} pairs as several allocation of similar size having the same stack trace may be recorded only once in the database. We emulate different stack traces by calling different entries in pfnArr. Each call then leads to foo20 but the resulting stack trace depth is different. We also enabled “Create user mode stack trace database” checkbox in gflags.exe for our application called AllocFree.exe.
Then we see the expansion of Stack Trace Database regions (addresses are different because memory dumps were taken from different application runs):
0:000> !address
[...]
+ 0`00240000 0`00312000 0`000d2000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database]
0`00312000 0`01a37000 0`01725000 MEM_PRIVATE MEM_RESERVE Other [Stack Trace Database]
0`01a37000 0`01a40000 0`00009000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database]
0:000> !address
[...]
+ 0`001b0000 0`0188c000 0`016dc000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database]
0`0188c000 0`0188d000 0`00001000 MEM_PRIVATE MEM_RESERVE Other [Stack Trace Database]
0`0188d000 0`019b0000 0`00123000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database]
Heap stays the same:
0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x000000f841c4f9c0
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock
Fast
(k) (k) (k) (k) length blocks cont.
heap
-------------------------------------------------------------------------------
------
0000000001a40000 08000002 4096 1444 4096 1164 4 3 0 0
LFH
External fragmentation 80 % (4 free blocks)
0000000000010000 08008000 64 4 64 1 1 1 0 0
0000000000020000 08008000 64 64 64 61 1 1 0 0
-------------------------------------------------------------------------------
------
0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x000000473a639107
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock
Fast
(k) (k) (k) (k) length blocks cont.
heap
-------------------------------------------------------------------------------
------
00000000019c0000 08000002 4096 1444 4096 1164 4 3 0 0
LFH
External fragmentation 80 % (4 free blocks)
0000000000010000 08008000 64 4 64 1 1 1 0 0
0000000000020000 08008000 64 64 64 61 1 1 0 0
-------------------------------------------------------------------------------
------
But we see the thread consuming much CPU and caught while logging stack backtrace:
0:000> kc
Call Site
ntdll!RtlpStdLogCapturedStackTrace
ntdll!RtlStdLogStackTrace
ntdll!RtlLogStackBackTraceEx
ntdll!RtlpAllocateHeap
ntdll!RtlAllocateHeap
AllocFree!_heap_alloc
AllocFree!malloc
AllocFree!foo20
AllocFree!foo19
AllocFree!foo18
AllocFree!foo17
AllocFree!foo16
AllocFree!foo15
AllocFree!foo14
AllocFree!foo13
AllocFree!foo12
AllocFree!foo11
AllocFree!foo10
AllocFree!foo9
AllocFree!foo8
AllocFree!foo7
AllocFree!foo6
AllocFree!foo5
AllocFree!foo4
AllocFree!foo3
AllocFree!foo2
AllocFree!foo1
AllocFree!wmain
AllocFree!__tmainCRTStartup
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
0:000> !runaway f
User Mode Time
Thread Time
0:53b8 0 days 3:22:02.354
Kernel Mode Time
Thread Time
0:53b8 0 days 0:20:39.022
Elapsed Time
Thread Time
0:53b8 0 days 10:11:23.596
If we dump some portion of the region we see recorded stack backtraces:
0:000> dps 0`0188c000-200 L200/8
00000000`0188be00 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd
00000000`0188be08 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c
00000000`0188be10 00000001`3fcc13cb AllocFree!malloc+0x5b
00000000`0188be18 00000001`3fcc1015 AllocFree!foo20+0x15
00000000`0188be20 00000001`3fcc1041 AllocFree!foo19+0x11
00000000`0188be28 00000001`3fcc1061 AllocFree!foo18+0x11
00000000`0188be30 00000001`3fcc12e3 AllocFree!wmain+0x53
00000000`0188be38 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144
00000000`0188be40 00000000`777259ed kernel32!BaseThreadInitThunk+0xd
00000000`0188be48 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d
00000000`0188be50 00000000`0188b1d0
00000000`0188be58 0009457d`00024fff
00000000`0188be60 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd
00000000`0188be68 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c
00000000`0188be70 00000001`3fcc13cb AllocFree!malloc+0x5b
00000000`0188be78 00000001`3fcc1015 AllocFree!foo20+0x15
00000000`0188be80 00000001`3fcc1041 AllocFree!foo19+0x11
00000000`0188be88 00000001`3fcc12e3 AllocFree!wmain+0x53
00000000`0188be90 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144
00000000`0188be98 00000000`777259ed kernel32!BaseThreadInitThunk+0xd
00000000`0188bea0 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d
00000000`0188bea8 00000000`00000000
00000000`0188beb0 00000000`0188b230
00000000`0188beb8 0008457e`00023fff
00000000`0188bec0 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd
00000000`0188bec8 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c
00000000`0188bed0 00000001`3fcc13cb AllocFree!malloc+0x5b
00000000`0188bed8 00000001`3fcc1015 AllocFree!foo20+0x15
00000000`0188bee0 00000001`3fcc12e3 AllocFree!wmain+0x53
00000000`0188bee8 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144
00000000`0188bef0 00000000`777259ed kernel32!BaseThreadInitThunk+0xd
00000000`0188bef8 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d
00000000`0188bf00 00000000`0188b280
00000000`0188bf08 001b457f`0002dfff
00000000`0188bf10 00000000`77891142 ntdll!RtlpAllocateHeap+0x33bd
00000000`0188bf18 00000000`778834d8 ntdll!RtlAllocateHeap+0x16c
00000000`0188bf20 00000001`3fcc13cb AllocFree!malloc+0x5b
00000000`0188bf28 00000001`3fcc1015 AllocFree!foo20+0x15
00000000`0188bf30 00000001`3fcc1041 AllocFree!foo19+0x11
00000000`0188bf38 00000001`3fcc1061 AllocFree!foo18+0x11
00000000`0188bf40 00000001`3fcc1081 AllocFree!foo17+0x11
00000000`0188bf48 00000001`3fcc10a1 AllocFree!foo16+0x11
00000000`0188bf50 00000001`3fcc10c1 AllocFree!foo15+0x11
00000000`0188bf58 00000001`3fcc10e1 AllocFree!foo14+0x11
00000000`0188bf60 00000001`3fcc1101 AllocFree!foo13+0x11
00000000`0188bf68 00000001`3fcc1121 AllocFree!foo12+0x11
00000000`0188bf70 00000001`3fcc1141 AllocFree!foo11+0x11
00000000`0188bf78 00000001`3fcc1161 AllocFree!foo10+0x11
00000000`0188bf80 00000001`3fcc1181 AllocFree!foo9+0x11
00000000`0188bf88 00000001`3fcc11a1 AllocFree!foo8+0x11
00000000`0188bf90 00000001`3fcc11c1 AllocFree!foo7+0x11
00000000`0188bf98 00000001`3fcc11e1 AllocFree!foo6+0x11
00000000`0188bfa0 00000001`3fcc1201 AllocFree!foo5+0x11
00000000`0188bfa8 00000001`3fcc1221 AllocFree!foo4+0x11
00000000`0188bfb0 00000001`3fcc1241 AllocFree!foo3+0x11
00000000`0188bfb8 00000001`3fcc1261 AllocFree!foo2+0x11
00000000`0188bfc0 00000001`3fcc1281 AllocFree!foo1+0x11
00000000`0188bfc8 00000001`3fcc12e3 AllocFree!wmain+0x53
00000000`0188bfd0 00000001`3fcc156c AllocFree!__tmainCRTStartup+0x144
00000000`0188bfd8 00000000`777259ed kernel32!BaseThreadInitThunk+0xd
00000000`0188bfe0 00000000`7785c541 ntdll!RtlUserThreadStart+0x1d
00000000`0188bfe8 00000000`00000000
00000000`0188bff0 00000000`00000000
00000000`0188bff8 00000000`00000000
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Models, Software Defect Construction, x64 Windows | No Comments »
Saturday, April 28th, 2012
Sometimes I hear voices saying that Linux, FreeBSD, and Mac OS X core dumps are uninteresting. This is not true. If you haven’t seen anything interesting there it just simply means you have only encountered a limited amount of abnormal software behaviour. The widespread usage of Windows OS means that most patterns have been diagnosed and described first and other OS are waiting their turn.
My goal is to have a pattern catalog with examples from different OS. For example, currently, all Mac OS X patterns I provide are just examples to existing Windows pattern names. All OS share the same structure and behavior, for example, structural memory analysis patterns and the same computational model. Although structural patterns are different from behavioral patterns I also plan to expand the structural list significantly especially in relation to forthcoming Windows malware analysis training. Regarding behavioral patterns it is possible to model and predict specific pattern examples for another OS by using already existing catalog.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Linux Crash Corner, Mac Crash Corner, Mac OS X, Malware Analysis, Malware Patterns, Pattern Models, Pattern Prediction, Pattern-Driven Debugging, Pattern-Driven Software Support, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Diagnostics | No Comments »
Wednesday, September 21st, 2011
Memoretics promotes pattern-driven memory dump and software trace analysis which has many uses but not limited to:
- Software and site reliability
- Software Debugging
- QA and Software Testing
- Computer Security
- Software Troubleshooting
- Malware Research and Analysis
- Tools as a Service (TaaS)
- Supportability
- Software Diagnostics

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Cloud Memory Dump Analysis, Computer Forensics, Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Trends, Escalation Engineering, Malware Analysis, Malware Patterns, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Metadefect Template Library, Metamalware, Pattern Models, Pattern Prediction, Reverse Engineering, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Defect Construction, Software Engineering, Software Generalist, Software Narratology, Software Support Patterns, Software Technical Support, Software Trace Analysis, Software Troubleshooting Patterns, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Victimware, Windows System Administration, Workaround Patterns | No Comments »
Wednesday, September 21st, 2011
Memoretics as a science of memory snapshots borrows many ideas from the following disciplines (the list is not exhaustive):
- Troubleshooting and Debugging
- Intelligence Analysis
- Critical Thinking
- Forensics
- Linguistics
- Archaeology
- Psychoanalysis
- History
- Mathematics: Sets and Categories
- Literary Criticism and Narratology
It also contributes many ideas back. The following diagram depicts such an interaction:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Archaeology of Computer Memory, Categorical Debugging, Computer Forensics, Computer Science, Countefactual Debugging, Crash Dump Analysis, Crash Dump De-analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Dublin School of Security, Education and Research, Escalation Engineering, General Memory Analysis, Generative Debugging, Hermeneutics of Memory Dumps and Traces, History, Intelligence Analysis Patterns, M->analysis, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Memory Dump Analysis and History, Memory Systems Language, Memory Visualization, Metadefect Template Library, Metamalware, New Debugging School, Paleo-debugging, Pattern Models, Pattern Prediction, Psychoanalysis of Software Maintenance and Support, Psychology, Reverse Engineering, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Linguistics, Software Trace Reading, Software Trace Visualization, Software Tracing Implementation Patterns, Software Troubleshooting Patterns, Software Victimology, Software and History, Structural Memory Analysis and Social Sciences, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Victimware, Virtualization, Visual Dump Analysis, Workaround Patterns | No Comments »
Thursday, July 14th, 2011
As a part of unified debugging pattern and generative debugging approach we extend software behavior analysis patterns such as memory dump and software trace analysis with UI abnormal behaviour patterns. Here by abnormality we mean behavior that users should not encounter while using software. Typical example is some error message or GUI distortion during execution of a functional use case. Such patterns will extend software behavior analysis pattern language we use for description of various post-construction software problems.
The first pattern we start with is called Error Message Box and we link it to Message Box and Self-Diagnosis memory analysis patterns. You can download x86 and x64 modeling examples from this location:
UIPMessageBox.zip
When we start the application it shows a message box:

We then launch Task Manager and find the window:

Then we save a crash dump using right-click context menu:

When we open the process memory dump we see this stack trace:
0:000> ~*kL
. 0 Id: d30.71c Suspend: 0 Teb: 000007ff`fffdd000 Unfrozen
Child-SP RetAddr Call Site
00000000`002ff1e8 00000000`77837214 user32!ZwUserWaitMessage+0xa
00000000`002ff1f0 00000000`778374a5 user32!DialogBox2+0x274
00000000`002ff280 00000000`778827f0 user32!InternalDialogBox+0x135
00000000`002ff2e0 00000000`77881ae5 user32!SoftModalMessageBox+0x9b4
00000000`002ff410 00000000`7788133b user32!MessageBoxWorker+0x31d
00000000`002ff5d0 00000000`77881232 user32!MessageBoxTimeoutW+0xb3
00000000`002ff6a0 00000001`3ffa101d user32!MessageBoxW+0×4e
00000000`002ff6e0 00000001`3ffa1039 UIPMessageBox!bar+0×1d
00000000`002ff710 00000001`3ffa1052 UIPMessageBox!foo+0×9
00000000`002ff740 00000001`3ffa11ea UIPMessageBox!wmain+0×12
00000000`002ff770 00000000`7770f56d UIPMessageBox!__tmainCRTStartup+0×15a
00000000`002ff7b0 00000000`77942cc1 kernel32!BaseThreadInitThunk+0xd
00000000`002ff7e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
We see there that foo function called bar function which displayed the message box. In real scenarios function name could me more meaningful and give a clue for troubleshooting and debugging in addition to message text:
0:000> ub 00000001`3ffa101d
UIPMessageBox!__unguarded_readlc_active+0xfff:
00000001`3ffa0fff add byte ptr [rax-7Dh],cl
00000001`3ffa1002 in al,dx
00000001`3ffa1003 sub byte ptr [rbp+33h],al
00000001`3ffa1006 leave
00000001`3ffa1007 lea r8,[UIPMessageBox!__mnames+0×28 (00000001`3ffa83c8)]
00000001`3ffa100e lea rdx,[UIPMessageBox!__mnames+0×38 (00000001`3ffa83d8)]
00000001`3ffa1015 xor ecx,ecx
00000001`3ffa1017 call qword ptr [UIPMessageBox!_imp_MessageBoxW (00000001`3ffa71d8)]
0:000> du 00000001`3ffa83c8
00000001`3ffa83c8 “Problem”
0:000> du 00000001`3ffa83d8
00000001`3ffa83d8 “We have a problem!”
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Crash Dump Analysis, Crash Dump Patterns, Debugging, Generative Debugging, Pattern Models, Software Behavior Patterns, UI Problem Analysis Patterns, Unified Debugging Patterns, x64 Windows | No Comments »
Sunday, April 17th, 2011
I’m pleased to announce that MDAA, Volume 5 is available in PDF format:
www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+5
It features:
- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4
Its table of contents is available here:
www.dumpanalysis.org/MDAA/MDA-Anthology-V5-TOC.pdf
Paperback and hardcover versions should be available in a week or two. I also started working on Volume 6 that should be available in November-December.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Aesthetics of Memory Dumps, Analysis Notation, Announcements, AntiPatterns, Archaeology of Computer Memory, Art, Assembly Language, Best Practices, Books, C and C++, CDF Analysis Tips and Tricks, Categorical Debugging, Citrix, Common Mistakes, Common Questions, Complete Memory Dump Analysis, Computer Forensics, Computer Science, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump De-analysis, Crash Dump Patterns, Crash Dumps for Dummies, Cyber Warfare, Debugging, Debugging Bureau, Debugging Industry, Debugging Methodology, Debugging Slang, Debugging Trends, Deep Down C++, Dr. Watson, Dublin School of Security, Education and Research, Escalation Engineering, Fun with Crash Dumps, Fun with Debugging, Fun with Software Traces, General Memory Analysis, Hermeneutics of Memory Dumps and Traces, Images of Computer Memory, Kernel Development, Malware Analysis, Mathematics of Debugging, Memiotics (Memory Semiotics), Memory Analysis Forensics and Intelligence, Memory Diagrams, Memory Dump Analysis Services, Memory Dumps in Myths, Memory Space Art, Memory Systems Language, Memory Visualization, Memory and Glitches, Metaphysics of Memory Worldview, Multithreading, Music for Debugging, New Acronyms, New Debugging School, New Words, Pattern Models, Philosophy, Physicalist Art, Publishing, Reverse Engineering, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Behavior Patterns, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Reading, Software Trace Visualization, Software Tracing for Dummies, Software Troubleshooting Patterns, Software Victimology, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, The Way of Philip Marlowe, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Victimware, Vista, Webinars, WinDbg Scripting Extensions, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns, x64 Windows | No Comments »
Monday, March 7th, 2011

The first Webinar to start an in-depth discussion of pattern-driven software troubleshooting, debugging and maintenance:
Date: 25th of March 2011
Time: 18:30 (GMT) 14:30 (EST) 11:30 (PST)
Duration: 60 minutes
Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/448268158
Topics include:
- A Short History of DumpAnalysis.org
- Memory Dump Analysis Patterns
- Troubleshooting and Debugging Tools (Debugware) Patterns
- Software Trace Analysis Patterns
- From Software Defects to Software Behavior
- Workaround Patterns
- Structural Memory Patterns
- Memory Analysis Domain Pattern Hierarchy
- New Directions
Prerequisites: experience in software troubleshooting and/or debugging.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Analysis Notation, Announcements, AntiPatterns, Best Practices, CDA Pattern Classification, Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Methodology, Debugging Trends, Escalation Engineering, Java Debugging, Linux Crash Corner, Mac Crash Corner, Malware Analysis, Malware Patterns, Memory Analysis Forensics and Intelligence, Memory Dump Analysis Services, Pattern Models, Pattern Prediction, Presentations, Software Behavior Patterns, Software Chorology, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Reading, Software Tracing Implementation Patterns, Software Troubleshooting Patterns, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Unified Debugging Patterns, Webinars, Workaround Patterns | No Comments »
Friday, December 17th, 2010
As the new decade is approaching (2011-2020) we would like to make a few previews and predictions:
- Increased complexity of software will bring more methods from biological, social sciences and humanities in addition to existing methods of automated debugging and computer science techniques
- Focus on first fault software problem solving (when aspect)
- Focus on pattern-driven software problem solving (how aspect)
- Fusion of debugging and malware analysis into a unified structural and behavioral pattern framework
- Visual debugging, memory and software trace visualization techniques
- Software maintenance certification
- Focus on domain-driven troubleshooting and debugging tools as a service (debugware TaaS)
- Focus on security issues related to memory dumps and software traces
- New scripting languages and programming language extensions for debugging
- The maturation of the science of memory snapshots and software traces (memoretics)
Imagining is not not limited to the above and more to come and explain in the forthcoming parts.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Anthropology, Archaeology of Computer Memory, Biology, Certification, Computer Science, Countefactual Debugging, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Industry, Debugging Methodology, Debugging Trends, Dublin School of Security, EasyDbg, Economics, Escalation Engineering, Evolution, First Fault Problem Solving, Futuristic Memory Dump Analysis, General Memory Analysis, General Science, Geography, Hardware, Hermeneutics of Memory Dumps and Traces, History, Humanities, Language, Laws of Troubleshooting and Debugging, Logic, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Memory Analysis Report System, Memory Dump Analysis and History, Memory Systems Language, Memory Visualization, Paleo-debugging, Pattern Models, Pattern Prediction, Physics, Psychology, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Social Media, Social Sciences, Software Architecture, Software Behavior Patterns, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narrative Fiction, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Linguistics, Software Trace Visualization, Software Troubleshooting Patterns, Software Victimology, Statistics, Structural Memory Analysis and Social Sciences, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, Unified Debugging Patterns, Uses of UML, Victimware, Virtualization, Visual Dump Analysis, Windows System Administration, Workaround Patterns | No Comments »
Friday, November 12th, 2010
Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca
The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:
- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4
Product information:
- Title: Memory Dump Analysis Anthology, Volume 5
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 400 pages
- Publisher: Opentask (10 December 2010)
- ISBN-13: 978-1-906717-96-4
- Hardcover: 400 pages
- Publisher: Opentask (10 December 2010)
- ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Aesthetics of Memory Dumps, Announcements, Archaeology of Computer Memory, Art, Assembly Language, Books, C and C++, CDF Analysis Tips and Tricks, Categorical Debugging, Common Mistakes, Complete Memory Dump Analysis, Computer Science, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump De-analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Debugging Slang, Deep Down C++, Dr. Watson, Dublin School of Security, Education and Research, Escalation Engineering, Fun with Crash Dumps, Fun with Debugging, Fun with Software Traces, General Memory Analysis, Hermeneutics of Memory Dumps and Traces, Images of Computer Memory, Kernel Development, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Analysis Report System, Memory Diagrams, Memory Dreams, Memory Dump Analysis Jobs, Memory Dump Analysis Services, Memory Dump Analysis and History, Memory Dumps in Movies, Memory Dumps in Myths, Memory Religion (Memorianity), Memory Space Art, Memory Systems Language, Memory Visualization, Memory and Glitches, Memuonics, Metaphysical Society of Ireland, Minidump Analysis, Movies and Debugging, Multithreading, Museum of Debugging, Music for Debugging, Music of Computation, New Acronyms, New Words, Paleo-debugging, Pattern Models, Pattern Prediction, Philosophy, Physicalist Art, Psychoanalysis of Software Maintenance and Support, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Behavior Patterns, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Reading, Software Trace Visualization, Software Tracing for Dummies, Software Troubleshooting Patterns, Software Victimology, Stack Trace Collection, Structural Memory Analysis and Social Sciences, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Theology, Tool Objects, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Uses of UML, Victimware, Virtualization, Vista, Visual Dump Analysis, Webinars, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns, x64 Windows | No Comments »
Sunday, July 18th, 2010

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free webinar
Date: 18th of August 2010
Time: 21:00 (BST) 16:00 (Eastern) 13:00 (Pacific)
Duration: 90 minutes
Topics include:
- User vs. kernel vs. physical (complete) memory space
- Challenges of complete memory dump analysis
- Common WinDbg commands
- Patterns
- Common mistakes
- Fiber bundles
- Hands-on exercise: a complete memory dump analysis
- A guide to DumpAnalysis.org case studies
Prerequisites: working knowledge of basic user process and kernel memory dump analysis or live debugging using WinDbg
The webinar link will be posted before 18th of August on DumpAnalysis.com
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Common Mistakes, Common Questions, Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Escalation Engineering, Memory Dump Analysis Services, Pattern Models, Security, Software Architecture, Software Behavior Patterns, Software Defect Construction, Software Engineering, Software Technical Support, Stack Trace Collection, Testing, Tools, Training and Seminars, Troubleshooting Methodology, Virtualization, Vista, Webinars, WinDbg Scripts, WinDbg Tips and Tricks, Windows 7, Windows Server 2008, Windows System Administration, x64 Windows | 1 Comment »
Tuesday, July 13th, 2010
Here we model Message Hooks pattern using MessageHistory tool. It uses window message hooking mechanism to intercept window messages. Download the tool and run either MessageHistory.exe or MessageHistory64.exe and push its Start button. Whenever any process becomes active after that either mhhooks.dll or mhhooks64.dll gets injected into the process virtual address space. Then we run WinDbg x86 or WinDbg x64, run notepad.exe and attach the debugger noninvasively to it:
*** wait with pending attach
Symbol search path is: srv*
Executable search path is:
WARNING: Process 2932 is not attached as a debuggee
The process can be examined but debug events will not be received
(b74.f44): Wake debugger - code 80000007 (first chance)
USER32!NtUserGetMessage+0xa:
00000000`76f9c92a c3 ret
0:000> .symfix
0:000> .reload
0:000> k
Child-SP RetAddr Call Site
00000000`0028f908 00000000`76f9c95e USER32!NtUserGetMessage+0xa
00000000`0028f910 00000000`ff511064 USER32!GetMessageW+0x34
00000000`0028f940 00000000`ff51133c notepad!WinMain+0x182
00000000`0028f9c0 00000000`76e7f56d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`0028fa80 00000000`770b3281 kernel32!BaseThreadInitThunk+0xd
00000000`0028fab0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
If we don’t select “Noninvasive” in “Attach to Process” dialog box we need to switch from the debugger injected thread to our main notepad application thread:
0:001> .symfix
0:001> .reload
0:001> k
Child-SP RetAddr Call Site
00000000`024bfe18 00000000`77178638 ntdll!DbgBreakPoint
00000000`024bfe20 00000000`76e7f56d ntdll!DbgUiRemoteBreakin+0x38
00000000`024bfe50 00000000`770b3281 kernel32!BaseThreadInitThunk+0xd
00000000`024bfe80 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
0:001> ~0s
USER32!NtUserGetMessage+0xa:
00000000`76f9c92a c3 ret
0:000> k
Child-SP RetAddr Call Site
00000000`000af9e8 00000000`76f9c95e USER32!NtUserGetMessage+0xa
00000000`000af9f0 00000000`ff511064 USER32!GetMessageW+0x34
00000000`000afa20 00000000`ff51133c notepad!WinMain+0x182
00000000`000afaa0 00000000`76e7f56d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`000afb60 00000000`770b3281 kernel32!BaseThreadInitThunk+0xd
00000000`000afb90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
We then inspect the raw stack data to see any execution residue and find a few related function calls:
0:000> !teb
TEB at 000007fffffdd000
ExceptionList: 0000000000000000
StackBase: 0000000000290000
StackLimit: 000000000027f000
SubSystemTib: 0000000000000000
FiberData: 0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self: 000007fffffdd000
EnvironmentPointer: 0000000000000000
ClientId: 0000000000000b74 . 0000000000000f44
RpcHandle: 0000000000000000
Tls Storage: 000007fffffdd058
PEB Address: 000007fffffdf000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
0:000> dps 000000000027f000 0000000000290000
[...]
00000000`0028e388 00000000`008bd8e0
00000000`0028e390 00000000`00000000
00000000`0028e398 00000000`00000001
00000000`0028e3a0 00000000`00000282
00000000`0028e3a8 00000000`76f966b2 USER32!SendMessageToUI+0x6a
00000000`0028e3b0 00000000`001406b0
00000000`0028e3b8 00000000`004000f8
00000000`0028e3c0 00000000`00000001
00000000`0028e3c8 00000001`800014b8 mhhooks64!CallWndProc+0×2d8
00000000`0028e3d0 00000000`00000000
00000000`0028e3d8 00000000`002f0664
00000000`0028e3e0 00000000`00000001
00000000`0028e3e8 00000000`76f96a72 USER32!ImeNotifyHandler+0xb4
00000000`0028e3f0 00000000`00000000
00000000`0028e3f8 00000000`004000f8
00000000`0028e400 00000000`00000001
00000000`0028e408 000007fe`ff1213b4 IMM32!CtfImmDispatchDefImeMessage+0×60
00000000`0028e410 00000000`00000000
00000000`0028e418 00000000`002f0664
00000000`0028e420 00000000`00000000
00000000`0028e428 00000000`002f0664
00000000`0028e430 00000000`008bd8e0
00000000`0028e438 00000000`76f96a06 USER32!ImeWndProcWorker+0×3af
00000000`0028e440 00000000`00000282
00000000`0028e448 00000000`00000000
00000000`0028e450 00000000`00000001
00000000`0028e458 00000000`004000f8
00000000`0028e460 00000000`00000000
00000000`0028e468 00000000`00000001
00000000`0028e470 00000000`00000000
00000000`0028e478 00000000`00000000
00000000`0028e480 00000000`00000000
00000000`0028e488 00000000`76f9a078 USER32!_fnDWORD+0×44
00000000`0028e490 00000000`00000000
[…]
00000000`0028f770 00000000`001406b0
00000000`0028f778 000007ff`fffdd000
00000000`0028f780 00000000`0028f8c8
00000000`0028f788 00000000`008bd8e0
00000000`0028f790 00000000`00000018
00000000`0028f798 00000000`76f885a0 USER32!DispatchHookW+0×2c
00000000`0028f7a0 000022b2`00000000
00000000`0028f7a8 00000000`00000001
00000000`0028f7b0 000007fe`ff2d2560 MSCTF!IMCLock::`vftable’
00000000`0028f7b8 00000000`00407c50
00000000`0028f7c0 00000000`000c0e51
00000000`0028f7c8 00000000`00000000
00000000`0028f7d0 00000000`00000000
00000000`0028f7d8 00000000`00000113
00000000`0028f7e0 00000000`00000113
00000000`0028f7e8 00000000`00000001
00000000`0028f7f0 00000000`00000000
00000000`0028f7f8 00000000`76f9c3df USER32!UserCallWinProcCheckWow+0×1cb
00000000`0028f800 00000000`ff510000 notepad!CFileDialogEvents_QueryInterface <PERF> (notepad+0×0)
00000000`0028f808 00000000`00000000
00000000`0028f810 00000000`00000000
00000000`0028f818 00000000`00000000
00000000`0028f820 00000000`00000000
00000000`0028f828 00000000`00000038
00000000`0028f830 00000000`00000000
00000000`0028f838 00000000`00000000
00000000`0028f840 00000000`00000000
00000000`0028f848 00000000`770cfdf5 ntdll!KiUserCallbackDispatcherContinue
00000000`0028f850 00000000`00000048
00000000`0028f858 00000000`00000001
00000000`0028f860 00000000`00000000
[…]
We also see a 3rd-party module in proximity having “hook” in its module name: mhhooks64. We disassemble its address to see yet another message hooking evidence:
0:000> .asm no_code_bytes
Assembly options: no_code_bytes
0:000> ub 00000001`800014b8
mhhooks64!CallWndProc+0×2ae:
00000001`8000148e imul rcx,rcx,30h
00000001`80001492 lea rdx,[mhhooks64!sendMessages (00000001`80021030)]
00000001`80001499 mov dword ptr [rdx+rcx+28h],eax
00000001`8000149d mov r9,qword ptr [rsp+50h]
00000001`800014a2 mov r8,qword ptr [rsp+48h]
00000001`800014a7 mov edx,dword ptr [rsp+40h]
00000001`800014ab mov rcx,qword ptr [mhhooks64!hCallWndHook (00000001`80021028)]
00000001`800014b2 call qword ptr [mhhooks64!_imp_CallNextHookEx (00000001`80017280)]
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Pattern Models, Software Behavior Patterns | No Comments »
Sunday, May 23rd, 2010
We continue our modeling of software behaviour with the ubiquitous Memory Leak (process heap) pattern. Instead of leaking small heap allocations that are easy to debug with user mode stack trace database our model program leaks large heap allocations:
// MemoryLeak-ProcessHeap
// Copyright (c) 2010 Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
// create extra 25 heaps initially
for (int i = 0; i < 25; ++i)
HeapCreate(0, 0, 0);
// create a heap to leak within
HANDLE hHeap = HeapCreate(0, 0, 0);
while (true)
{
HeapAlloc(hHeap, 0, 1024*1024);
Sleep(1000);
}
return 0;
}
The program creates extra process heaps to simulate real life heap leaks that usually happen not in a default process heap. Then it slowly leaks 0×100000 bytes every second. The application can be downloaded from here (zip file contains source code, x86 and x64 binaries together with corresponding PDB files):
Download MemoryLeak-ProcessHeap.zip
Here I present the results from x64 Windows Server 2008 R2 but x86 variants (I tested on x86 Vista) should be the same.
First we run the application and save a dump of it after a few seconds (I used Task Manager). Heap statistics shows 9 virtual blocks for the last 0000000001e00000 heap:
0:000> !heap -s
LFH Key : 0x000000d529c37801
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
————————————————————————————-
00000000002b0000 00000002 1024 164 1024 3 1 1 0 0 LFH
0000000000010000 00008000 64 4 64 1 1 1 0 0
0000000000020000 00008000 64 64 64 61 1 1 0 0
0000000000220000 00001002 1088 152 1088 3 2 2 0 0 LFH
0000000000630000 00001002 512 8 512 3 1 1 0 0
0000000000870000 00001002 512 8 512 3 1 1 0 0
0000000000ad0000 00001002 512 8 512 3 1 1 0 0
00000000007e0000 00001002 512 8 512 3 1 1 0 0
0000000000cc0000 00001002 512 8 512 3 1 1 0 0
0000000000ed0000 00001002 512 8 512 3 1 1 0 0
00000000010c0000 00001002 512 8 512 3 1 1 0 0
00000000005b0000 00001002 512 8 512 3 1 1 0 0
00000000009f0000 00001002 512 8 512 3 1 1 0 0
00000000004d0000 00001002 512 8 512 3 1 1 0 0
0000000000230000 00001002 512 8 512 3 1 1 0 0
0000000000700000 00001002 512 8 512 3 1 1 0 0
00000000012d0000 00001002 512 8 512 3 1 1 0 0
0000000000950000 00001002 512 8 512 3 1 1 0 0
0000000000b90000 00001002 512 8 512 3 1 1 0 0
00000000014c0000 00001002 512 8 512 3 1 1 0 0
0000000000e50000 00001002 512 8 512 3 1 1 0 0
0000000001020000 00001002 512 8 512 3 1 1 0 0
00000000016e0000 00001002 512 8 512 3 1 1 0 0
0000000001940000 00001002 512 8 512 3 1 1 0 0
0000000001b90000 00001002 512 8 512 3 1 1 0 0
0000000001200000 00001002 512 8 512 3 1 1 0 0
0000000000c20000 00001002 512 8 512 3 1 1 0 0
0000000000db0000 00001002 512 8 512 3 1 1 0 0
0000000000f50000 00001002 512 8 512 3 1 1 0 0
Virtual block: 0000000001350000 - 0000000001350000 (size 0000000000000000)
Virtual block: 0000000001540000 - 0000000001540000 (size 0000000000000000)
Virtual block: 0000000001760000 - 0000000001760000 (size 0000000000000000)
Virtual block: 00000000019c0000 - 00000000019c0000 (size 0000000000000000)
Virtual block: 0000000001c10000 - 0000000001c10000 (size 0000000000000000)
Virtual block: 0000000001e80000 - 0000000001e80000 (size 0000000000000000)
Virtual block: 0000000001f90000 - 0000000001f90000 (size 0000000000000000)
Virtual block: 00000000020a0000 - 00000000020a0000 (size 0000000000000000)
Virtual block: 00000000021b0000 - 00000000021b0000 (size 0000000000000000)
0000000001e00000 00001002 512 8 512 3 1 1 9 0
————————————————————————————-
We then wait for a few minutes and save a memory dump again. Heap statistics clearly shows virtual block leaks because now we 276 of them instead of previous 9 (I skipped most of them in the output below):
0:000> !heap -s
LFH Key : 0x000000d529c37801
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
————————————————————————————-
00000000002b0000 00000002 1024 164 1024 3 1 1 0 0 LFH
0000000000010000 00008000 64 4 64 1 1 1 0 0
0000000000020000 00008000 64 64 64 61 1 1 0 0
0000000000220000 00001002 1088 152 1088 3 2 2 0 0 LFH
0000000000630000 00001002 512 8 512 3 1 1 0 0
0000000000870000 00001002 512 8 512 3 1 1 0 0
0000000000ad0000 00001002 512 8 512 3 1 1 0 0
00000000007e0000 00001002 512 8 512 3 1 1 0 0
0000000000cc0000 00001002 512 8 512 3 1 1 0 0
0000000000ed0000 00001002 512 8 512 3 1 1 0 0
00000000010c0000 00001002 512 8 512 3 1 1 0 0
00000000005b0000 00001002 512 8 512 3 1 1 0 0
00000000009f0000 00001002 512 8 512 3 1 1 0 0
00000000004d0000 00001002 512 8 512 3 1 1 0 0
0000000000230000 00001002 512 8 512 3 1 1 0 0
0000000000700000 00001002 512 8 512 3 1 1 0 0
00000000012d0000 00001002 512 8 512 3 1 1 0 0
0000000000950000 00001002 512 8 512 3 1 1 0 0
0000000000b90000 00001002 512 8 512 3 1 1 0 0
00000000014c0000 00001002 512 8 512 3 1 1 0 0
0000000000e50000 00001002 512 8 512 3 1 1 0 0
0000000001020000 00001002 512 8 512 3 1 1 0 0
00000000016e0000 00001002 512 8 512 3 1 1 0 0
0000000001940000 00001002 512 8 512 3 1 1 0 0
0000000001b90000 00001002 512 8 512 3 1 1 0 0
0000000001200000 00001002 512 8 512 3 1 1 0 0
0000000000c20000 00001002 512 8 512 3 1 1 0 0
0000000000db0000 00001002 512 8 512 3 1 1 0 0
0000000000f50000 00001002 512 8 512 3 1 1 0 0
Virtual block: 0000000001350000 - 0000000001350000 (size 0000000000000000)
Virtual block: 0000000001540000 - 0000000001540000 (size 0000000000000000)
Virtual block: 0000000001760000 - 0000000001760000 (size 0000000000000000)
Virtual block: 00000000019c0000 - 00000000019c0000 (size 0000000000000000)
[… skipped …]
Virtual block: 00000000131b0000 - 00000000131b0000 (size 0000000000000000)
Virtual block: 00000000132c0000 - 00000000132c0000 (size 0000000000000000)
Virtual block: 00000000133d0000 - 00000000133d0000 (size 0000000000000000)
Virtual block: 00000000134e0000 - 00000000134e0000 (size 0000000000000000)
Virtual block: 00000000135f0000 - 00000000135f0000 (size 0000000000000000)
Virtual block: 0000000013700000 - 0000000013700000 (size 0000000000000000)
Virtual block: 0000000013810000 - 0000000013810000 (size 0000000000000000)
Virtual block: 0000000013920000 - 0000000013920000 (size 0000000000000000)
Virtual block: 0000000013a30000 - 0000000013a30000 (size 0000000000000000)
Virtual block: 0000000013b40000 - 0000000013b40000 (size 0000000000000000)
Virtual block: 0000000013c50000 - 0000000013c50000 (size 0000000000000000)
Virtual block: 0000000013d60000 - 0000000013d60000 (size 0000000000000000)
0000000001e00000 00001002 512 8 512 3 1 1 276 0
————————————————————————————-
We see that size of these blocks is 0×101000 bytes (with hindsight, extra 1000 is probably bookkeeping info):
0:000> !address 0000000013d60000
ProcessParametrs 00000000002b1f20 in range 00000000002b0000 00000000002d9000
Environment 00000000002b1320 in range 00000000002b0000 00000000002d9000
0000000013d60000 : 0000000013d60000 - 0000000000101000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageHeap
Handle 0000000001e00000
We want to know which thread allocates them and we search for the heap address 0000000001e00000 through virtual memory to find any execution residue on thread raw stacks:
0:000> s -q 0 LFFFFFF 0000000001e00000
00000000`001cf608 00000000`01e00000 00000000`01e00000
00000000`001cf610 00000000`01e00000 00000000`00000858
00000000`001cf630 00000000`01e00000 00000000`0000000a
00000000`001cf6c8 00000000`01e00000 00000000`00000000
00000000`001cf6e0 00000000`01e00000 00000000`01e00a80
00000000`001cf720 00000000`01e00000 00000000`00000020
00000000`001cf778 00000000`01e00000 00000000`01e00000
00000000`001cf780 00000000`01e00000 00000000`00000000
00000000`001cf798 00000000`01e00000 00000000`01e02000
00000000`001cf7b0 00000000`01e00000 02100301`00000000
00000000`001cf7c8 00000000`01e00000 00000000`01c10000
00000000`001cf808 00000000`01e00000 00000000`00000001
00000000`001cf830 00000000`01e00000 00000000`00000002
00000000`001cf940 00000000`01e00000 00000000`00000000
00000000`002d8378 00000000`01e00000 004c0044`005c0064
00000000`01e00028 00000000`01e00000 00000000`01e00000
00000000`01e00030 00000000`01e00000 00000000`00000080
Address range 00000000`001cfxxxx belongs to the main thread:
0:000> kL
Child-SP RetAddr Call Site
00000000`001cf898 000007fe`fdd91203 ntdll!NtDelayExecution+0xa
00000000`001cf8a0 00000001`3f39104f KERNELBASE!SleepEx+0xab
00000000`001cf940 00000001`3f3911ea MemoryLeak_ProcessHeap!wmain+0×4f
00000000`001cf970 00000000`778cf56d MemoryLeak_ProcessHeap!__tmainCRTStartup+0×15a
00000000`001cf9b0 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd
00000000`001cf9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
We dump the raw stack fragment now:
0:000> dps 00000000`001cf608 00000000`001cf940
00000000`001cf608 00000000`01e00000
00000000`001cf610 00000000`01e00000
00000000`001cf618 00000000`00000858
00000000`001cf620 00000000`00000000
00000000`001cf628 00000000`77b229ac ntdll!RtlAllocateHeap+0×16c
00000000`001cf630 00000000`01e00000
00000000`001cf638 00000000`0000000a
00000000`001cf640 00000000`00000858
00000000`001cf648 00000000`00000860
00000000`001cf650 00000000`00000000
00000000`001cf658 00000000`001cf740
00000000`001cf660 00000020`00001000
00000000`001cf668 fffff680`01000000
00000000`001cf670 00000001`3f390000 MemoryLeak_ProcessHeap!wmain <PERF> (MemoryLeak_ProcessHeap+0×0)
00000000`001cf678 01000000`00000080
00000000`001cf680 00000000`0000f000
00000000`001cf688 02100210`02100210
00000000`001cf690 00000001`3f390000 MemoryLeak_ProcessHeap!wmain <PERF> (MemoryLeak_ProcessHeap+0×0)
00000000`001cf698 00000000`0000f000
00000000`001cf6a0 00000000`01e01fd0
00000000`001cf6a8 00000000`77b07ff3 ntdll!RtlpCreateUCREntry+0xb3
00000000`001cf6b0 00000000`001cf6b8
00000000`001cf6b8 00000000`01e01fc0
00000000`001cf6c0 00000000`00000080
00000000`001cf6c8 00000000`01e00000
00000000`001cf6d0 00000000`00000000
00000000`001cf6d8 00000000`00000a00
00000000`001cf6e0 00000000`01e00000
00000000`001cf6e8 00000000`01e00a80
00000000`001cf6f0 00000000`0007e000
00000000`001cf6f8 00000000`77b0f2bb ntdll!RtlpInitializeHeapSegment+0×19b
00000000`001cf700 00000000`01e00208
00000000`001cf708 00000000`00000000
00000000`001cf710 00000000`01e00230
00000000`001cf718 00000000`00000000
00000000`001cf720 00000000`01e00000
00000000`001cf728 00000000`00000020
00000000`001cf730 00000000`00000000
00000000`001cf738 00000000`77b0f676 ntdll!RtlpInitializeUCRIndex+0×36
00000000`001cf740 000007ff`00000003
00000000`001cf748 00000000`00000100
00000000`001cf750 00000000`00000000
00000000`001cf758 00000000`00001002
00000000`001cf760 00000000`00001002
00000000`001cf768 00000000`77b0fec9 ntdll!RtlCreateHeap+0×8f7
00000000`001cf770 00000000`01e02000
00000000`001cf778 00000000`01e00000
00000000`001cf780 00000000`01e00000
00000000`001cf788 00000000`00000000
00000000`001cf790 03010301`00000000
00000000`001cf798 00000000`01e00000
00000000`001cf7a0 00000000`01e02000
00000000`001cf7a8 00000000`01e80000
00000000`001cf7b0 00000000`01e00000
00000000`001cf7b8 02100301`00000000
00000000`001cf7c0 00000000`001f0000
00000000`001cf7c8 00000000`01e00000
00000000`001cf7d0 00000000`01c10000
00000000`001cf7d8 00000000`01e02000
00000000`001cf7e0 00000000`00270000
00000000`001cf7e8 03020302`00000230
00000000`001cf7f0 00000000`77be7288 ntdll!RtlpInterceptorRoutines
00000000`001cf7f8 00000000`00000000
00000000`001cf800 00000000`00100010
00000000`001cf808 00000000`01e00000
00000000`001cf810 00000000`00000001
00000000`001cf818 00000000`00100000
00000000`001cf820 00000000`00000000
00000000`001cf828 00000000`77b229ac ntdll!RtlAllocateHeap+0×16c
00000000`001cf830 00000000`01e00000
00000000`001cf838 00000000`00000002
00000000`001cf840 00000000`00100000
00000000`001cf848 00000000`00101000
00000000`001cf850 00000000`00000000
00000000`001cf858 00000000`001cf940
00000000`001cf860 00000000`00000000
00000000`001cf868 0000f577`2bd1e0ff
00000000`001cf870 00000000`ffffffff
00000000`001cf878 00000000`10010011
00000000`001cf880 00000000`c00000bb
00000000`001cf888 00000000`00000000
00000000`001cf890 00000000`00000100
00000000`001cf898 000007fe`fdd91203 KERNELBASE!SleepEx+0xab
00000000`001cf8a0 00000000`001cf958
00000000`001cf8a8 00000000`00000000
00000000`001cf8b0 00000000`00000000
00000000`001cf8b8 00000000`00000012
00000000`001cf8c0 ffffffff`ff676980
00000000`001cf8c8 00000000`001cf8c0
00000000`001cf8d0 00000000`00000048
00000000`001cf8d8 00000000`00000001
00000000`001cf8e0 00000000`00000000
00000000`001cf8e8 00000000`00000000
00000000`001cf8f0 00000000`00000000
00000000`001cf8f8 00000000`00000000
00000000`001cf900 00000000`00000000
00000000`001cf908 00000000`00000000
00000000`001cf910 00000000`00000000
00000000`001cf918 00000000`00000000
00000000`001cf920 00000000`00000000
00000000`001cf928 00000000`00000001
00000000`001cf930 00000000`00000000
00000000`001cf938 00000001`3f39104f MemoryLeak_ProcessHeap!wmain+0×4f
00000000`001cf940 00000000`01e00000
We see traces of RtlAllocateHeap but it could be a coincidence (imagine that we don’t have access to source code and can’t do live debugging to put breakpoints). We advise to enable use mode stack trace database as explained for another example process: CTX106970. Then we launch our application again and save a new user dump. We repeat the same procedure to examine the raw stack:
0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x000000c21e1b31e6
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000001bc0000 08000002 1024 168 1024 5 1 1 0 0 LFH
0000000000010000 08008000 64 4 64 1 1 1 0 0
0000000000020000 08008000 64 64 64 61 1 1 0 0
0000000000100000 08001002 1088 152 1088 2 2 2 0 0 LFH
0000000001d90000 08001002 512 8 512 3 1 1 0 0
0000000001f90000 08001002 512 8 512 3 1 1 0 0
00000000021c0000 08001002 512 8 512 3 1 1 0 0
0000000002130000 08001002 512 8 512 3 1 1 0 0
0000000002370000 08001002 512 8 512 3 1 1 0 0
0000000001e80000 08001002 512 8 512 3 1 1 0 0
0000000000110000 08001002 512 8 512 3 1 1 0 0
0000000002510000 08001002 512 8 512 3 1 1 0 0
0000000002760000 08001002 512 8 512 3 1 1 0 0
0000000001cc0000 08001002 512 8 512 3 1 1 0 0
0000000002030000 08001002 512 8 512 3 1 1 0 0
0000000002960000 08001002 512 8 512 3 1 1 0 0
0000000002670000 08001002 512 8 512 3 1 1 0 0
0000000002b90000 08001002 512 8 512 3 1 1 0 0
00000000022f0000 08001002 512 8 512 3 1 1 0 0
00000000028b0000 08001002 512 8 512 3 1 1 0 0
0000000001f10000 08001002 512 8 512 3 1 1 0 0
0000000002450000 08001002 512 8 512 3 1 1 0 0
00000000025f0000 08001002 512 8 512 3 1 1 0 0
0000000002a40000 08001002 512 8 512 3 1 1 0 0
0000000002c90000 08001002 512 8 512 3 1 1 0 0
0000000002d90000 08001002 512 8 512 3 1 1 0 0
0000000002e80000 08001002 512 8 512 3 1 1 0 0
0000000002fc0000 08001002 512 8 512 3 1 1 0 0
00000000030b0000 08001002 512 8 512 3 1 1 0 0
Virtual block: 0000000003130000 - 0000000003130000 (size 0000000000000000)
Virtual block: 0000000003240000 - 0000000003240000 (size 0000000000000000)
Virtual block: 0000000003350000 - 0000000003350000 (size 0000000000000000)
Virtual block: 0000000003460000 - 0000000003460000 (size 0000000000000000)
Virtual block: 0000000003570000 - 0000000003570000 (size 0000000000000000)
Virtual block: 0000000003680000 - 0000000003680000 (size 0000000000000000)
Virtual block: 0000000003790000 - 0000000003790000 (size 0000000000000000)
Virtual block: 00000000038a0000 - 00000000038a0000 (size 0000000000000000)
Virtual block: 00000000039b0000 - 00000000039b0000 (size 0000000000000000)
Virtual block: 0000000003ac0000 - 0000000003ac0000 (size 0000000000000000)
Virtual block: 0000000003bd0000 - 0000000003bd0000 (size 0000000000000000)
Virtual block: 0000000003ce0000 - 0000000003ce0000 (size 0000000000000000)
0000000002270000 08001002 512 8 512 3 1 1 12 0
0:000> s -q 0 LFFFFFF 0000000002270000
00000000`0029f648 00000000`02270000 00000000`000000a8
00000000`0029f660 00000000`02270000 00000000`77b99bc7
00000000`0029f6e8 00000000`02270000 00000000`02270000
00000000`0029f6f0 00000000`02270000 00000000`00000858
00000000`0029f710 00000000`02270000 00000000`0000000a
00000000`0029f758 00000000`02270000 00000000`0029f918
00000000`0029f7c0 00000000`02270000 00000000`03ce0040
00000000`0029f858 00000000`02270000 00000000`02270000
00000000`0029f860 00000000`02270000 00000000`00000000
00000000`0029f878 00000000`02270000 00000000`02272000
00000000`0029f890 00000000`02270000 02100301`00000000
00000000`0029f8f8 00000000`02270000 00000000`03ce0040
00000000`0029fa20 00000000`02270000 00000000`00000000
00000000`01be8a08 00000000`02270000 0064006e`00690057
00000000`02270028 00000000`02270000 00000000`02270000
00000000`02270030 00000000`02270000 00000000`00000080
0:000> dqs 00000000`0029f648 00000000`0029fa20
00000000`0029f648 00000000`02270000
00000000`0029f650 00000000`000000a8
00000000`0029f658 00000000`00000a80
00000000`0029f660 00000000`02270000
00000000`0029f668 00000000`77b99bc7 ntdll!RtlStdLogStackTrace+0x47
00000000`0029f670 00000000`002a0000
00000000`0029f678 00000000`0000007e
00000000`0029f680 00000000`02270a80
00000000`0029f688 00000000`00000000
00000000`0029f690 00000000`02270208
00000000`0029f698 00070000`77b99bc7
00000000`0029f6a0 00000000`77b6cd8a ntdll! ?? ::FNODOBFM::`string'+0xf35a
00000000`0029f6a8 00000000`77b0fd07 ntdll!RtlCreateHeap+0x56e
00000000`0029f6b0 000007fe`fdd9c6a4 KERNELBASE!HeapCreate+0x54
00000000`0029f6b8 00000001`3faa1030 MemoryLeak_ProcessHeap!wmain+0x30
00000000`0029f6c0 00000001`3faa11ea MemoryLeak_ProcessHeap!__tmainCRTStartup+0x15a
00000000`0029f6c8 00000000`778cf56d kernel32!BaseThreadInitThunk+0xd
00000000`0029f6d0 00000000`77be7288 ntdll!RtlpInterceptorRoutines
00000000`0029f6d8 00000000`00000000
00000000`0029f6e0 00000000`00000860
00000000`0029f6e8 00000000`02270000
00000000`0029f6f0 00000000`02270000
00000000`0029f6f8 00000000`00000858
00000000`0029f700 00000000`00000000
00000000`0029f708 00000000`77b229ac ntdll!RtlAllocateHeap+0x16c
00000000`0029f710 00000000`02270000
00000000`0029f718 00000000`0000000a
00000000`0029f720 00000000`00000858
00000000`0029f728 00000000`00000860
00000000`0029f730 00000000`00000000
00000000`0029f738 00000000`0029f820
00000000`0029f740 00000000`77be7288 ntdll!RtlpInterceptorRoutines
00000000`0029f748 00000000`00000002
00000000`0029f750 00000000`00100030
00000000`0029f758 00000000`02270000
00000000`0029f760 00000000`0029f918
00000000`0029f768 00000000`00000020
00000000`0029f770 00000000`00000002
00000000`0029f778 00000000`00000005
00000000`0029f780 00000000`000750f0
00000000`0029f788 00000000`77ba25b2 ntdll!RtlpRegisterStackTrace+0x92
00000000`0029f790 00000000`000750b8
00000000`0029f798 00000000`00000003
00000000`0029f7a0 00000000`00000000
00000000`0029f7a8 00000000`77ad7a0a ntdll!RtlCaptureStackBackTrace+0x4a
00000000`0029f7b0 00000000`00000002
00000000`0029f7b8 00000000`00100030
00000000`0029f7c0 00000000`02270000
00000000`0029f7c8 00000000`03ce0040
00000000`0029f7d0 00000000`00100020
00000000`0029f7d8 00000000`77ba2eb7 ntdll!RtlpStackTraceDatabaseLogPrefix+0x57
00000000`0029f7e0 00000000`03ce0040
00000000`0029f7e8 00000000`00000000
00000000`0029f7f0 00000000`00100020
00000000`0029f7f8 00000000`000750f0
00000000`0029f800 00000000`77b6ed2d ntdll! ?? ::FNODOBFM::`string’+0×1a81b
00000000`0029f808 00000001`3faa1044 MemoryLeak_ProcessHeap!wmain+0×44
00000000`0029f810 00000001`3faa11ea MemoryLeak_ProcessHeap!__tmainCRTStartup+0×15a
00000000`0029f818 00000000`778cf56d kernel32!BaseThreadInitThunk+0xd
00000000`0029f820 00000000`77b03281 ntdll!RtlUserThreadStart+0×1d
00000000`0029f828 00000000`00000100
00000000`0029f830 00000000`00000000
00000000`0029f838 00000000`08001002
00000000`0029f840 00000000`08001002
00000000`0029f848 00000000`77b0fec9 ntdll!RtlCreateHeap+0×8f7
00000000`0029f850 00000000`02272000
00000000`0029f858 00000000`02270000
00000000`0029f860 00000000`02270000
00000000`0029f868 00000000`00000000
00000000`0029f870 03010301`00000000
00000000`0029f878 00000000`02270000
00000000`0029f880 00000000`02272000
00000000`0029f888 00000000`022f0000
00000000`0029f890 00000000`02270000
00000000`0029f898 02100301`00000000
00000000`0029f8a0 00000000`00001000
00000000`0029f8a8 00000000`77b9a886 ntdll!RtlpSetupExtendedBlock+0xc6
00000000`0029f8b0 00000000`00000000
00000000`0029f8b8 00000000`02272000
00000000`0029f8c0 00000000`000b0000
00000000`0029f8c8 03020302`00000230
00000000`0029f8d0 00000000`77be7288 ntdll!RtlpInterceptorRoutines
00000000`0029f8d8 00000000`00000002
00000000`0029f8e0 00000000`77be7288 ntdll!RtlpInterceptorRoutines
00000000`0029f8e8 00000000`00000002
00000000`0029f8f0 00000000`00100030
00000000`0029f8f8 00000000`02270000
00000000`0029f900 00000000`03ce0040
00000000`0029f908 00000000`77b6ed6a ntdll! ?? ::FNODOBFM::`string’+0×1a858
00000000`0029f910 00000000`00000000
00000000`0029f918 00000000`00000000
00000000`0029f920 00000000`00100000
00000000`0029f928 00000000`00101000
00000000`0029f930 00000000`00000020
00000000`0029f938 00000000`00000002
00000000`0029f940 00000000`00000000
00000000`0029f948 0000f569`df709780
00000000`0029f950 00000000`ffffffff
00000000`0029f958 00000000`12010013
00000000`0029f960 00000000`c00000bb
00000000`0029f968 00000000`00000000
00000000`0029f970 00000000`00000100
00000000`0029f978 000007fe`fdd91203 KERNELBASE!SleepEx+0xab
00000000`0029f980 00000000`0029fa38
00000000`0029f988 00000000`00000000
00000000`0029f990 00000000`00000000
00000000`0029f998 00000000`00000012
00000000`0029f9a0 ffffffff`ff676980
00000000`0029f9a8 00000000`0029f9a0
00000000`0029f9b0 00000000`00000048
00000000`0029f9b8 00000000`00000001
00000000`0029f9c0 00000000`00000000
00000000`0029f9c8 00000000`00000000
00000000`0029f9d0 00000000`00000000
00000000`0029f9d8 00000000`00000000
00000000`0029f9e0 00000000`00000000
00000000`0029f9e8 00000000`00000000
00000000`0029f9f0 00000000`00000000
00000000`0029f9f8 00000000`00000000
00000000`0029fa00 00000000`00000000
00000000`0029fa08 00000000`00000001
00000000`0029fa10 00000000`00000000
00000000`0029fa18 00000001`3faa104f MemoryLeak_ProcessHeap!wmain+0×4f
00000000`0029fa20 00000000`02270000
Now we see this stack trace fragment from user mode stack trace database on the raw stack shown above:
00000000`0029f800 00000000`77b6ed2d ntdll! ?? ::FNODOBFM::`string'+0x1a81b
00000000`0029f808 00000001`3faa1044MemoryLeak_ProcessHeap!wmain+0×44
00000000`0029f810 00000001`3faa11ea MemoryLeak_ProcessHeap!__tmainCRTStartup+0×15a
00000000`0029f818 00000000`778cf56d kernel32!BaseThreadInitThunk+0xd
00000000`0029f820 00000000`77b03281 ntdll!RtlUserThreadStart+0×1d
And it looks like HeapAlloc was called from wmain indeed with 0×100000 parameter:
0:000> ub 00000001`3faa1044
MemoryLeak_ProcessHeap!wmain+0x26:
00000001`3faa1026 xor edx,edx
00000001`3faa1028 xor ecx,ecx
00000001`3faa102a call qword ptr [MemoryLeak_ProcessHeap!_imp_HeapCreate (00000001`3faa7000)]
00000001`3faa1030 mov rbx,rax
00000001`3faa1033 xor edx,edx
00000001`3faa1035 mov r8d,100000h
00000001`3faa103b mov rcx,rbx
00000001`3faa103e call qword ptr [MemoryLeak_ProcessHeap!_imp_HeapAlloc (00000001`3faa7008)]
0:000> dps 00000001`3faa7008 L1
00000001`3faa7008 00000000`77b21b70 ntdll!RtlAllocateHeap
The stack trace fragment from x86 Vista user dump even more straightforward:
0040fa00 77946e0c ntdll!RtlAllocateHeap+0×1e3
0040fa04 0022103a MemoryLeak_ProcessHeap!wmain+0×3a
0040fa08 7677d0e9 kernel32!BaseThreadInitThunk+0xe
0040fa0c 779219bb ntdll!__RtlUserThreadStart+0×23
0040fa10 7792198e ntdll!_RtlUserThreadStart+0×1b
0040fa14 7798924f ntdll!RtlpLogCapturedStackTrace+0×103
Of course, we could simply disassemble wmain after identifying our thread but in real life functions are longer and leaking allocations could have happened from frames not present on current stack traces.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Pattern Models, Software Behavior Patterns, Software Defect Construction | No Comments »
Sunday, May 16th, 2010
Due to many requests for memory dumps corresponding to crash dump analysis patterns I’ve started modeling software behaviour and defects. Every pattern will have an example application(s), service(s) or driver(s) or combination of them. Their execution results in memory layout that corresponds to memory or trace analysis patterns. Today we introduce an example model for Multiple Exceptions (user mode) pattern. The following source code models 3 threads each having an exception during their execution on Windows XP, Windows 7 and Windows Server 2008 R2:
// MultipleExceptions-UserMode
// Copyright (c) 2010 Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt
#include <windows.h>
#include <process.h>
void thread_one(void *)
{
*(int *)NULL = 0;
}
void thread_two(void *)
{
*(int *)NULL = 0;
}
int main(int argc, WCHAR* argv[])
{
_beginthread(thread_two, 0, NULL);
_beginthread(thread_one, 0, NULL);
DebugBreak();
return 0;
}
In fact, thread_one and thread_two can be replaced with just one function because they are identical. Visual C++ compiler does that during code optimization. On Windows 7 and W2K8 R2 I created LocalDumps registry key to save full crash dumps. On Windows XP I set Dr. Watson as a postmortem debugger (via drwtsn32 -i command and configured it to save full user dumps via drwtsn32 command that brings Dr. Watson GUI). Vista had some peculiar behaviour so I postponed its discussion for another post. The application can be downloaded from here (zip file contains source code, x86 and x64 binaries together with corresponding PDB files):
Download MultipleExceptions-UserMode.zip
Now I provide modeling results for x64 W2K8 R2 running on 2 processor machine. Windows 7 and Windows XP results are very similar. If we run x64 executable it crashes and a dump file is saved (x86 crash dump is similar). Default analysis command gives these results:
0:000> !analyze -v
[...]
FAULTING_IP:
MultipleExceptions_UserMode!thread_two+0
00000001`3f8b1000 c704250000000000000000 mov dword ptr [0],0
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007fefddc2442 (KERNELBASE!DebugBreak+0x0000000000000002)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 0000000000000000
[...]
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
[...]
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT
[...]
STACK_TEXT:
00000001`3f8b1000 MultipleExceptions_UserMode!thread_two+0x0
00000001`3f8b10eb MultipleExceptions_UserMode!_callthreadstart+0x17
00000001`3f8b1195 MultipleExceptions_UserMode!_threadstart+0x95
00000000`778cf56d kernel32!BaseThreadInitThunk+0xd
00000000`77b03281 ntdll!RtlUserThreadStart+0x1d
[...]
We see debug break on the first thread:
0:000> kL
Child-SP RetAddr Call Site
00000000`002eec78 000007fe`fdd913a6 ntdll!NtWaitForMultipleObjects+0xa
00000000`002eec80 00000000`778d3143 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`002eed80 00000000`77949025 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`002eee10 00000000`779491a7 kernel32!WerpReportFaultInternal+0x215
00000000`002eeeb0 00000000`779491ff kernel32!WerpReportFault+0x77
00000000`002eeee0 00000000`7794941c kernel32!BasepReportFault+0x1f
00000000`002eef10 00000000`77b6573c kernel32!UnhandledExceptionFilter+0x1fc
00000000`002eeff0 00000000`77ae5148 ntdll! ?? ::FNODOBFM::`string'+0x2365
00000000`002ef020 00000000`77b0554d ntdll!_C_specific_handler+0x8c
00000000`002ef090 00000000`77ae5d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`002ef0c0 00000000`77b1fe48 ntdll!RtlDispatchException+0x3cb
00000000`002ef7a0 000007fe`fddc2442 ntdll!KiUserExceptionDispatcher+0x2e
00000000`002efd58 00000001`3f8b103c KERNELBASE!DebugBreak+0×2
00000000`002efd60 00000001`3f8b13fb MultipleExceptions_UserMode!main+0×2c
00000000`002efd90 00000000`778cf56d MultipleExceptions_UserMode!__tmainCRTStartup+0×15b
00000000`002efdd0 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd
00000000`002efe00 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
2 other threads show exception processing too:
0:000> ~1s; kL
ntdll!NtDelayExecution+0xa:
00000000`77b201fa c3 ret
Child-SP RetAddr Call Site
00000000`0076ef78 000007fe`fdd91203 ntdll!NtDelayExecution+0xa
00000000`0076ef80 00000000`77949175 KERNELBASE!SleepEx+0xab
00000000`0076f020 00000000`779491ff kernel32!WerpReportFault+0×45
00000000`0076f050 00000000`7794941c kernel32!BasepReportFault+0×1f
00000000`0076f080 00000000`77b6573c kernel32!UnhandledExceptionFilter+0×1fc
00000000`0076f160 00000000`77ae5148 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`0076f190 00000000`77b0554d ntdll!_C_specific_handler+0×8c
00000000`0076f200 00000000`77ae5d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`0076f230 00000000`77b1fe48 ntdll!RtlDispatchException+0×3cb
00000000`0076f910 00000001`3f8b1000 ntdll!KiUserExceptionDispatcher+0×2e
00000000`0076fec8 00000001`3f8b10eb MultipleExceptions_UserMode!thread_two
00000000`0076fed0 00000001`3f8b1195 MultipleExceptions_UserMode!_callthreadstart+0×17
00000000`0076ff00 00000000`778cf56d MultipleExceptions_UserMode!_threadstart+0×95
00000000`0076ff30 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd
00000000`0076ff60 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
0:001> ~2s; kL
ntdll!NtDelayExecution+0xa:
00000000`77b201fa c3 ret
Child-SP RetAddr Call Site
00000000`0086e968 000007fe`fdd91203 ntdll!NtDelayExecution+0xa
00000000`0086e970 00000000`77949175 KERNELBASE!SleepEx+0xab
00000000`0086ea10 00000000`779491ff kernel32!WerpReportFault+0×45
00000000`0086ea40 00000000`7794941c kernel32!BasepReportFault+0×1f
00000000`0086ea70 00000000`77b6573c kernel32!UnhandledExceptionFilter+0×1fc
00000000`0086eb50 00000000`77ae5148 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`0086eb80 00000000`77b0554d ntdll!_C_specific_handler+0×8c
00000000`0086ebf0 00000000`77ae5d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`0086ec20 00000000`77b1fe48 ntdll!RtlDispatchException+0×3cb
00000000`0086f300 00000001`3f8b1000 ntdll!KiUserExceptionDispatcher+0×2e
00000000`0086f8b8 00000001`3f8b10eb MultipleExceptions_UserMode!thread_two
00000000`0086f8c0 00000001`3f8b1195 MultipleExceptions_UserMode!_callthreadstart+0×17
00000000`0086f8f0 00000000`778cf56d MultipleExceptions_UserMode!_threadstart+0×95
00000000`0086f920 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd
00000000`0086f950 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
We look at unhandled exception filter parameter to get exception pointers information:
0:002> kv
Child-SP RetAddr : Args to Child : Call Site
[...]
00000000`0086ea70 00000000`77b6573c : 00000000`0086ebb0 00000000`00000006 00000001`00000000 00000000`00000001 : kernel32!UnhandledExceptionFilter+0×1fc
0:002> .exptr 00000000`0086ebb0
----- Exception record at 00000000`0086f7f0:
ExceptionAddress: 000000013f8b1000 (MultipleExceptions_UserMode!thread_two)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000
----- Context record at 00000000`0086f300:
rax=00000000000ef0b0 rbx=00000000000ef0b0 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=000000013f8b1000 rsp=000000000086f8b8 rbp=0000000000000000
r8=000007fffffda000 r9=0000000000000000 r10=0000000000000045
r11=000007fffffd9328 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
MultipleExceptions_UserMode!thread_two:
00000001`3f8b1000 c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=????????
What we now see that default analysis command showed the break instruction exception record and error code from the first thread but IP and stack trace from other threads having NULL pointer access violation exception.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Pattern Models, Software Behavior Patterns, Software Defect Construction | No Comments »