Crash Dump Analysis Patterns (Part 13e)
I’m coming back to my old Insufficient Memory pattern range (see Committed Memory, Handle Leak, Kernel Pool, PTE). This post discusses the user space case when we don’t have enough virtual memory available for reservation due to memory fragmentation. For example, a java virtual machine is pre-allocating memory for its garbage-collected heap. However after installing some 3rd-party software the amount of pre-allocated memory is less than expected. In such cases it is possible to do comparative memory dump analysis to see the difference in virtual address spaces. Original memory dump has this module distribution in memory:
0:000> lm
start end module name
00400000 0040b000 javaw (deferred)
009e0000 009e7000 hpi (deferred)
00a30000 00a3e000 verify (deferred)
00a40000 00a59000 java (deferred)
00a60000 00a6d000 zip (deferred)
03ff0000 03fff000 net (deferred)
040a0000 040a8000 nio (deferred)
040b0000 0410a000 hnetcfg (deferred)
041d0000 042e2000 awt (deferred)
04540000 04591000 fontmanager (deferred)
04620000 04670000 msctf (deferred)
047c0000 047de000 jpeg (deferred)
05820000 05842000 dcpr (deferred)
05920000 05932000 pkcs11wrapper (deferred)
08000000 08139000 jvm (deferred)
10000000 100e0000 moduleA (deferred)
68000000 68035000 rsaenh (deferred)
6e220000 6e226000 RMProcessLink (deferred)
71ae0000 71ae8000 wshtcpip (deferred)
71b20000 71b61000 mswsock (deferred)
71bf0000 71bf8000 ws2help (deferred)
71c00000 71c17000 ws2_32 (deferred)
71c20000 71c32000 tsappcmp (deferred)
71c40000 71c97000 netapi32 (deferred)
73070000 73097000 winspool (deferred)
76290000 762ad000 imm32 (deferred)
76920000 769e2000 userenv (deferred)
76aa0000 76acd000 winmm (deferred)
76b70000 76b7b000 psapi (deferred)
76ed0000 76efa000 dnsapi (deferred)
76f10000 76f3e000 wldap32 (deferred)
76f50000 76f63000 secur32 (deferred)
76f70000 76f77000 winrnr (deferred)
76f80000 76f85000 rasadhlp (deferred)
77380000 77411000 user32 (deferred)
77670000 777a9000 ole32 (deferred)
77ba0000 77bfa000 msvcrt (deferred)
77c00000 77c48000 gdi32 (deferred)
77c50000 77cef000 rpcrt4 (deferred)
77e40000 77f42000 kernel32 (deferred)
77f50000 77feb000 advapi32 (deferred)
78130000 781cb000 msvcr80 (deferred)
7c800000 7c8c0000 ntdll (pdb symbols)
We see the big gap between 100e0000 and 68000000 addresses. This means that it is theoretically possible to reserve and/or commit up to 57F20000 bytes (about 1.4Gb). !address WinDbg command shows that at least 1.1Gb region (shown in bold below) was reserved indeed:
0:000> !address
00000000 : 00000000 - 00010000
Type 00000000
Protect 00000001 PAGE_NOACCESS
State 00010000 MEM_FREE
Usage RegionUsageFree
00010000 : 00010000 - 00001000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageEnvironmentBlock
00011000 : 00011000 - 0000f000
Type 00000000
Protect 00000001 PAGE_NOACCESS
State 00010000 MEM_FREE
Usage RegionUsageFree
00020000 : 00020000 - 00001000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageProcessParametrs
00021000 : 00021000 - 0000f000
Type 00000000
Protect 00000001 PAGE_NOACCESS
State 00010000 MEM_FREE
Usage RegionUsageFree
00030000 : 00030000 - 00003000
Type 00020000 MEM_PRIVATE
Protect 00000140 <unk>
State 00001000 MEM_COMMIT
Usage RegionUsageStack
Pid.Tid 97c.1b3c
...
...
...
100e0000 : 100e0000 - 000a0000
Type 00020000 MEM_PRIVATE
Protect 00000040 PAGE_EXECUTE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
10180000 - 05cc0000
Type 00020000 MEM_PRIVATE
State 00002000 MEM_RESERVE
Usage RegionUsageIsVAD
15e40000 - 004f3000
Type 00020000 MEM_PRIVATE
Protect 00000040 PAGE_EXECUTE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
16333000 - 45bad000
Type 00020000 MEM_PRIVATE
State 00002000 MEM_RESERVE
Usage RegionUsageIsVAD
5bee0000 - 00ac0000
Type 00020000 MEM_PRIVATE
Protect 00000040 PAGE_EXECUTE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
5c9a0000 - 03540000
Type 00020000 MEM_PRIVATE
State 00002000 MEM_RESERVE
Usage RegionUsageIsVAD
5fee0000 : 5fee0000 - 00120000
Type 00000000
Protect 00000001 PAGE_NOACCESS
State 00010000 MEM_FREE
Usage RegionUsageFree
…
…
…
Looking at the problem memory dump we see that the gap is smaller (less than 1.1Gb):
0:000> lm
start end module name
00400000 0040b000 javaw (deferred)
08000000 08139000 jvm (deferred)
10000000 10007000 hpi (deferred)
51120000 511bb000 msvcr80 # (private pdb symbols)
520f0000 520fe000 verify (deferred)
52100000 52119000 java (deferred)
52120000 5212d000 zip (deferred)
52130000 5213f000 net (deferred)
52140000 52148000 nio (deferred)
52150000 52262000 awt (deferred)
52270000 522c1000 fontmanager (deferred)
522d0000 52320000 MSCTF (deferred)
52330000 5234e000 jpeg (deferred)
52350000 52372000 dcpr (deferred)
52510000 52522000 pkcs11wrapper (deferred)
5f270000 5f2ca000 hnetcfg (deferred)
60000000 60029000 3rdPartyHook (deferred)
61e80000 61e86000 detoured (export symbols)
68000000 68035000 rsaenh (deferred)
71ae0000 71ae8000 wshtcpip (deferred)
71b20000 71b61000 mswsock (deferred)
71bf0000 71bf8000 ws2help (deferred)
71c00000 71c17000 ws2_32 (deferred)
71c20000 71c32000 tsappcmp (deferred)
71c40000 71c97000 netapi32 (deferred)
73070000 73097000 winspool (deferred)
76290000 762ad000 imm32 (deferred)
76920000 769e2000 userenv (deferred)
76aa0000 76acd000 winmm (deferred)
76b70000 76b7b000 psapi (deferred)
76ed0000 76efa000 dnsapi (deferred)
76f10000 76f3e000 wldap32 (deferred)
76f50000 76f63000 secur32 (deferred)
76f70000 76f77000 winrnr (deferred)
76f80000 76f85000 rasadhlp (deferred)
77380000 77411000 user32 (pdb symbols)
77670000 777a9000 ole32 (deferred)
77ba0000 77bfa000 msvcrt (deferred)
77c00000 77c48000 gdi32 (deferred)
77c50000 77cef000 rpcrt4 (deferred)
77e40000 77f42000 kernel32 (pdb symbols)
77f50000 77feb000 advapi32 (pdb symbols)
7c340000 7c396000 msvcr71 (deferred)
7c800000 7c8c0000 ntdll (pdb symbols)
!address command shows that less memory was reserved in the latter case (about 896Mb):
0:000> !address
...
...
...
10010000 : 10010000 - 000a0000
Type 00020000 MEM_PRIVATE
Protect 00000040 PAGE_EXECUTE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
100b0000 - 04a70000
Type 00020000 MEM_PRIVATE
State 00002000 MEM_RESERVE
Usage RegionUsageIsVAD
14b20000 - 004a6000
Type 00020000 MEM_PRIVATE
Protect 00000040 PAGE_EXECUTE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
14fc6000 - 3804a000
Type 00020000 MEM_PRIVATE
State 00002000 MEM_RESERVE
Usage RegionUsageIsVAD
4d010000 - 00ac0000
Type 00020000 MEM_PRIVATE
Protect 00000040 PAGE_EXECUTE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
4dad0000 - 03540000
Type 00020000 MEM_PRIVATE
State 00002000 MEM_RESERVE
Usage RegionUsageIsVAD
51010000 : 51010000 - 00110000
Type 00000000
Protect 00000001 PAGE_NOACCESS
State 00010000 MEM_FREE
Usage RegionUsageFree
…
…
…
Looking at module list again we notice that most java runtime modules were shifted to 50000000 address range. We also notice that new 3rdPartyHook and detoured modules appear in our problem memory dump.
Module information is missing for detoured module:
0:000> lmv m detoured
start end module name
61e80000 61e86000 detoured (deferred)
Image path: C:\WINDOWS\system32\detoured.dll
Image name: detoured.dll
Timestamp: Thu Feb 07 04:14:16 2008 (47AA8598)
CheckSum: 0000EF91
ImageSize: 00006000
File version: 0.0.0.0
Product version: 0.0.0.0
File flags: 0 (Mask 0)
File OS: 0 Unknown Base
File type: 0.0 Unknown
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
However applying Unknown Component pattern we see that is Microsoft Research Detours Package:
0:000> db 61e80000 61e86000
61e80000 MZ..............
61e80010 ........@.......
61e80020 ................
61e80030 ................
61e80040 ........!..L.!Th
61e80050 is program canno
61e80060 t be run in DOS
61e80070 mode....$.......
61e80080 5...q...q...q...
61e80090 ....r...q...p...
61e800a0 V%..p...V%..p...
61e800b0 V%..p...V%..p...
61e800c0 Richq...........
61e800d0 ................
61e800e0 PE..L......G....
61e800f0 .......!........
61e80100 ................
...
...
...
...
61e84390 ..P.r.o.d.u.c.t.
61e843a0 N.a.m.e…..M.i.
61e843b0 c.r.o.s.o.f.t. .
61e843c0 R.e.s.e.a.r.c.h.
61e843d0 .D.e.t.o.u.r.s.
61e843e0 .P.a.c.k.a.g.e.
61e843f0 ….j.#…P.r.o.
61e84400 d.u.c.t.V.e.r.s.
61e84410 i.o.n…P.r.o.f.
61e84420 e.s.s.i.o.n.a.l.
61e84430 .V.e.r.s.i.o.n.
61e84440 .2…1. .B.u.i.
61e84450 l.d._.2.1.0…..
61e84460 D…..V.a.r.F.i.
…
…
…
0:000> du 61e843a0+C
61e843ac "Microsoft Research Detours Packa"
61e843ec "ge"
We can also see that 3rdPartyHook module imports this library and lots of kernel32 API related to memory allocation, file mapping and loading DLLs (see No Component Symbols pattern):
0:000> !dh 60000000
...
...
...
OPTIONAL HEADER VALUES
10B magic #
8.00 linker version
18000 size of code
F000 size of initialized data
0 size of uninitialized data
13336 address of entry point
1000 base of code
----- new -----
60000000 image base
1000 section alignment
1000 file alignment
2 subsystem (Windows GUI)
4.00 operating system version
0.00 image version
4.00 subsystem version
29000 size of image
1000 size of headers
3376F checksum
00100000 size of stack reserve
00001000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
0 [ 0] address [size] of Export Directory
218CC [ 8C] address [size] of Import Directory
25000 [ 5F4] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
28000 [ 19E0] address [size] of Security Directory
26000 [ 2670] address [size] of Base Relocation Directory
19320 [ 1C] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
1F3C0 [ 40] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
19000 [ 2B0] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory
…
…
…
0:000> dds 60000000+19000 60000000+19000+2B0
60019064 7c82b0dc ntdll!RtlReAllocateHeap
60019068 77e4ec39 kernel32!HeapDestroy
6001906c 77e41fba kernel32!GetSystemTimeAsFileTime
60019070 77e619d1 kernel32!GetTickCount
60019074 77e69577 kernel32!QueryPerformanceCounter
60019078 7c82a9be ntdll!RtlSizeHeap
6001907c 77e82060 kernel32!SetUnhandledExceptionFilter
60019080 77e7690d kernel32!UnhandledExceptionFilter
60019084 77e42004 kernel32!TerminateProcess
60019088 7c82a136 ntdll!RtlRestoreLastWin32Error
6001908c 77e77a5f kernel32!SuspendThread
60019090 77e76a26 kernel32!SetThreadContext
60019094 77e77ae3 kernel32!GetThreadContext
60019098 77e73347 kernel32!FlushInstructionCache
6001909c 77e5f38b kernel32!ResumeThread
600190a0 77e616a8 kernel32!InterlockedCompareExchange
600190a4 77e645a9 kernel32!VirtualAlloc
600190a8 77e41fe3 kernel32!VirtualProtect
600190ac 77e66ed1 kernel32!VirtualQuery
600190b0 77e44960 kernel32!GetLogicalDriveStringsA
600190b4 77eab401 kernel32!GetVolumeNameForVolumeMountPointA
600190b8 77e6794d kernel32!GetACP
600190bc 77e6f3cf kernel32!GetLocaleInfoA
600190c0 77e622b7 kernel32!GetThreadLocale
600190c4 77e69d74 kernel32!GetVersionExA
600190c8 77e4beab kernel32!RaiseException
600190cc 77e60037 kernel32!GetSystemDirectoryA
600190d0 77e52bf4 kernel32!GetWindowsDirectoryA
600190d4 77e5c7a8 kernel32!lstrcmpA
600190d8 77e46c99 kernel32!OutputDebugStringA
600190dc 77e5bd7d kernel32!CreateEventA
600190e0 77e62311 kernel32!SetEvent
600190e4 77e51281 kernel32!ExpandEnvironmentStringsA
600190e8 77e9f365 kernel32!MoveFileA
600190ec 77e5da00 kernel32!IsDebuggerPresent
600190f0 77e9e4b1 kernel32!QueryDosDeviceA
600190f4 7c829e08 ntdll!RtlGetLastWin32Error
600190f8 77e63d7a kernel32!GetProcAddress
600190fc 77e41dc6 kernel32!LoadLibraryA
60019100 7c829e17 ntdll!RtlFreeHeap
60019104 77e62419 kernel32!LocalFree
60019108 7c829fd6 ntdll!RtlAllocateHeap
6001910c 77e63ec7 kernel32!GetProcessHeap
60019110 77ea2186 kernel32!VerifyVersionInfoA
60019114 7c81379f ntdll!VerSetConditionMask
60019118 77e63143 kernel32!WideCharToMultiByte
6001911c 77e70550 kernel32!SizeofResource
60019120 77e6b11b kernel32!SetHandleCount
60019124 77e69bf9 kernel32!LoadResource
60019128 77e511e1 kernel32!FindResourceA
6001912c 77e7388c kernel32!FindResourceExA
60019130 77e5be30 kernel32!lstrlenA
60019134 77e424de kernel32!Sleep
60019138 77ea2cb1 kernel32!WaitNamedPipeA
6001913c 77e63e6f kernel32!CloseHandle
60019140 77e5e123 kernel32!OpenEventA
60019144 77e622c9 kernel32!lstrlenW
60019148 77e62fc7 kernel32!GetCurrentThreadId
6001914c 77e5fdd4 kernel32!OpenProcess
60019150 77e63c78 kernel32!GetCurrentProcessId
60019154 7c81a3ab ntdll!RtlLeaveCriticalSection
60019158 7c81a360 ntdll!RtlEnterCriticalSection
6001915c 77e4cabf kernel32!GetComputerNameA
60019160 77e6f032 kernel32!ProcessIdToSessionId
60019164 77e645ff kernel32!GetModuleFileNameA
60019168 77e6474a kernel32!GetModuleHandleA
6001916c 77e62f9d kernel32!GetCurrentProcess
60019170 77e49968 kernel32!GetCurrentDirectoryA
60019174 77e61c7b kernel32!WaitForSingleObject
60019178 77e63868 kernel32!GetCurrentThread
6001917c 7c82c988 ntdll!RtlDeleteCriticalSection
60019180 77e67861 kernel32!InitializeCriticalSection
60019184 77e6b1a1 kernel32!FreeLibrary
60019188 77e63f41 kernel32!UnmapViewOfFile
6001918c 77e643f1 kernel32!MapViewOfFile
60019190 77e6b65f kernel32!OpenFileMappingA
60019194 77e61694 kernel32!InterlockedExchange
60019198 77e4d2fb kernel32!DeleteFileA
...
...
...
60019294 00000000
60019298 61e81000 detoured!Detoured
6001929c 00000000
…
…
…
This warrants the suspicion that 3rdPartyHook somehow optimized the virtual address space for its own purposes and this resulted in more fragmented virtual address space.
- Dmitry Vostokov @ DumpAnalysis.org -
August 10th, 2008 at 11:26 am
[…] - Insufficient Memory (virtual memory) […]
February 17th, 2009 at 10:14 am
[…] Insufficient Memory (virtual memory) […]
June 10th, 2009 at 3:01 pm
[…] of Applicatione.exe and get 2-3 subsequent process memory dumps, when it grows, to inspect its virtual memory growth using differential memory analysis […]
June 25th, 2009 at 2:28 am
[…] box, find another hidden exception and from it see the real nested offender that experienced insufficient memory condition resulted in C++ exception. You can see WinDbg output from this dump in the post about […]
April 19th, 2010 at 10:54 am
[…] we introduce an icon for Insufficient Memory (virtual memory) […]
March 5th, 2012 at 11:36 pm
!address -summary WinDbg command also gives various output about region type usage and the largest available blocks per region type.
April 9th, 2012 at 2:36 pm
This pattern was renamed from “virtual memory” to “module fragmentation”