Crash Dump Analysis Patterns (Part 13e)

I’m coming back to my old Insufficient Memory pattern range (see Committed Memory, Handle Leak, Kernel Pool, PTE). This post discusses the user space case when we don’t have enough virtual memory available for reservation due to memory fragmentation. For example, a java virtual machine is pre-allocating memory for its garbage-collected heap. However after installing some 3rd-party software the amount of pre-allocated memory is less than expected. In such cases it is possible to do comparative memory dump analysis to see the difference in virtual address spaces. Original memory dump has this module distribution in memory:

0:000> lm
start    end        module name
00400000 0040b000   javaw      (deferred)            
009e0000 009e7000   hpi        (deferred)            
00a30000 00a3e000   verify     (deferred)            
00a40000 00a59000   java       (deferred)            
00a60000 00a6d000   zip        (deferred)            
03ff0000 03fff000   net        (deferred)            
040a0000 040a8000   nio        (deferred)            
040b0000 0410a000   hnetcfg    (deferred)            
041d0000 042e2000   awt        (deferred)            
04540000 04591000   fontmanager   (deferred)            
04620000 04670000   msctf      (deferred)            
047c0000 047de000   jpeg       (deferred)            
05820000 05842000   dcpr       (deferred)            
05920000 05932000   pkcs11wrapper   (deferred)            
08000000 08139000   jvm        (deferred)            
10000000 100e0000   moduleA    (deferred)            
68000000 68035000   rsaenh     (deferred)

6e220000 6e226000   RMProcessLink   (deferred)            
71ae0000 71ae8000   wshtcpip   (deferred)            
71b20000 71b61000   mswsock    (deferred)            
71bf0000 71bf8000   ws2help    (deferred)            
71c00000 71c17000   ws2_32     (deferred)            
71c20000 71c32000   tsappcmp   (deferred)            
71c40000 71c97000   netapi32   (deferred)            
73070000 73097000   winspool   (deferred)            
76290000 762ad000   imm32      (deferred)            
76920000 769e2000   userenv    (deferred)            
76aa0000 76acd000   winmm      (deferred)            
76b70000 76b7b000   psapi      (deferred)            
76ed0000 76efa000   dnsapi     (deferred)            
76f10000 76f3e000   wldap32    (deferred)            
76f50000 76f63000   secur32    (deferred)            
76f70000 76f77000   winrnr     (deferred)            
76f80000 76f85000   rasadhlp   (deferred)            
77380000 77411000   user32     (deferred)            
77670000 777a9000   ole32      (deferred)            
77ba0000 77bfa000   msvcrt     (deferred)            
77c00000 77c48000   gdi32      (deferred)            
77c50000 77cef000   rpcrt4     (deferred)            
77e40000 77f42000   kernel32   (deferred)            
77f50000 77feb000   advapi32   (deferred)            
78130000 781cb000   msvcr80    (deferred)            
7c800000 7c8c0000   ntdll      (pdb symbols)

We see the big gap between 100e0000 and 68000000 addresses. This means that it is theoretically possible to reserve and/or commit up to 57F20000 bytes (about 1.4Gb). !address WinDbg command shows that at least 1.1Gb region (shown in bold below) was reserved indeed:

0:000> !address
00000000 : 00000000 - 00010000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
00010000 : 00010000 - 00001000
              Type     00020000 MEM_PRIVATE
              Protect  00000004 PAGE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageEnvironmentBlock
00011000 : 00011000 - 0000f000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
00020000 : 00020000 - 00001000
              Type     00020000 MEM_PRIVATE
              Protect  00000004 PAGE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageProcessParametrs
00021000 : 00021000 - 0000f000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
00030000 : 00030000 - 00003000
              Type     00020000 MEM_PRIVATE
              Protect  00000140 <unk>
              State    00001000 MEM_COMMIT
              Usage    RegionUsageStack
              Pid.Tid  97c.1b3c
...
...
...
100e0000 : 100e0000 - 000a0000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         10180000 - 05cc0000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
         15e40000 - 004f3000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         16333000 - 45bad000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD

         5bee0000 - 00ac0000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         5c9a0000 - 03540000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
5fee0000 : 5fee0000 - 00120000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree


Looking at the problem memory dump we see that the gap is smaller (less than 1.1Gb):

0:000> lm
start    end        module name
00400000 0040b000   javaw      (deferred)            
08000000 08139000   jvm        (deferred)            
10000000 10007000   hpi        (deferred)            
51120000 511bb000   msvcr80  # (private pdb symbols)

520f0000 520fe000   verify     (deferred)            
52100000 52119000   java       (deferred)            
52120000 5212d000   zip        (deferred)            
52130000 5213f000   net        (deferred)            
52140000 52148000   nio        (deferred)            
52150000 52262000   awt        (deferred)            
52270000 522c1000   fontmanager   (deferred)            
522d0000 52320000   MSCTF      (deferred)            
52330000 5234e000   jpeg       (deferred)            
52350000 52372000   dcpr       (deferred)            
52510000 52522000   pkcs11wrapper   (deferred)            
5f270000 5f2ca000   hnetcfg    (deferred)            
60000000 60029000   3rdPartyHook   (deferred)            
61e80000 61e86000   detoured   (export symbols)
68000000 68035000   rsaenh     (deferred)            
71ae0000 71ae8000   wshtcpip   (deferred)            
71b20000 71b61000   mswsock    (deferred)            
71bf0000 71bf8000   ws2help    (deferred)            
71c00000 71c17000   ws2_32     (deferred)            
71c20000 71c32000   tsappcmp   (deferred)            
71c40000 71c97000   netapi32   (deferred)            
73070000 73097000   winspool   (deferred)            
76290000 762ad000   imm32      (deferred)            
76920000 769e2000   userenv    (deferred)            
76aa0000 76acd000   winmm      (deferred)            
76b70000 76b7b000   psapi      (deferred)            
76ed0000 76efa000   dnsapi     (deferred)            
76f10000 76f3e000   wldap32    (deferred)            
76f50000 76f63000   secur32    (deferred)            
76f70000 76f77000   winrnr     (deferred)            
76f80000 76f85000   rasadhlp   (deferred)            
77380000 77411000   user32     (pdb symbols)         
77670000 777a9000   ole32      (deferred)            
77ba0000 77bfa000   msvcrt     (deferred)            
77c00000 77c48000   gdi32      (deferred)            
77c50000 77cef000   rpcrt4     (deferred)            
77e40000 77f42000   kernel32   (pdb symbols)         
77f50000 77feb000   advapi32   (pdb symbols)         
7c340000 7c396000   msvcr71    (deferred)            
7c800000 7c8c0000   ntdll      (pdb symbols)

!address command shows that less memory was reserved in the latter case (about 896Mb):

0:000> !address
...
...
...
10010000 : 10010000 - 000a0000
             Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         100b0000 - 04a70000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
         14b20000 - 004a6000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         14fc6000 - 3804a000
             Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD

          4d010000 - 00ac0000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
          4dad0000 - 03540000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
51010000 : 51010000 - 00110000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree


Looking at module list again we notice that most java runtime modules were shifted to 50000000 address range. We also notice that new 3rdPartyHook and detoured modules appear in our problem memory dump. 

Module information is missing for detoured module:

0:000> lmv m detoured
start    end        module name
61e80000 61e86000   detoured   (deferred)            
    Image path: C:\WINDOWS\system32\detoured.dll
    Image name: detoured.dll
    Timestamp:        Thu Feb 07 04:14:16 2008 (47AA8598)
    CheckSum:         0000EF91
    ImageSize:        00006000
    File version:     0.0.0.0
    Product version:  0.0.0.0
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        0.0 Unknown
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

However applying Unknown Component pattern we see that is Microsoft Research Detours Package:

0:000> db 61e80000 61e86000
61e80000  MZ..............
61e80010  ........@.......
61e80020  ................
61e80030  ................
61e80040  ........!..L.!Th
61e80050  is program canno
61e80060  t be run in DOS
61e80070  mode....$.......
61e80080  5...q...q...q...
61e80090  ....r...q...p...
61e800a0  V%..p...V%..p...
61e800b0  V%..p...V%..p...
61e800c0  Richq...........
61e800d0  ................
61e800e0  PE..L......G....
61e800f0  .......!........
61e80100  ................
...
...
...
...
61e84390  ..P.r.o.d.u.c.t.
61e843a0  N.a.m.e…..M.i.
61e843b0  c.r.o.s.o.f.t. .
61e843c0  R.e.s.e.a.r.c.h.
61e843d0  .D.e.t.o.u.r.s.
61e843e0  .P.a.c.k.a.g.e.

61e843f0  ….j.#…P.r.o.
61e84400  d.u.c.t.V.e.r.s.
61e84410  i.o.n…P.r.o.f.
61e84420  e.s.s.i.o.n.a.l.
61e84430  .V.e.r.s.i.o.n.
61e84440  .2…1. .B.u.i.
61e84450  l.d._.2.1.0…..
61e84460  D…..V.a.r.F.i.


0:000> du 61e843a0+C
61e843ac  "Microsoft Research Detours Packa"
61e843ec  "ge"

We can also see that 3rdPartyHook module imports this library and lots of kernel32 API related to memory allocation, file mapping and loading DLLs (see No Component Symbols pattern):

0:000> !dh 60000000
...
...
...
OPTIONAL HEADER VALUES
     10B magic #
    8.00 linker version
   18000 size of code
    F000 size of initialized data
       0 size of uninitialized data
   13336 address of entry point
    1000 base of code
         ----- new -----
60000000 image base
    1000 section alignment
    1000 file alignment
       2 subsystem (Windows GUI)
    4.00 operating system version
    0.00 image version
    4.00 subsystem version
   29000 size of image
    1000 size of headers
   3376F checksum
00100000 size of stack reserve
00001000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
       0 [       0] address [size] of Export Directory
   218CC [      8C] address [size] of Import Directory
   25000 [     5F4] address [size] of Resource Directory
       0 [       0] address [size] of Exception Directory
   28000 [    19E0] address [size] of Security Directory
   26000 [    2670] address [size] of Base Relocation Directory
   19320 [      1C] address [size] of Debug Directory
       0 [       0] address [size] of Description Directory
       0 [       0] address [size] of Special Directory
       0 [       0] address [size] of Thread Storage Directory
   1F3C0 [      40] address [size] of Load Configuration Directory
       0 [       0] address [size] of Bound Import Directory
   19000 [     2B0] address [size] of Import Address Table Directory
       0 [       0] address [size] of Delay Import Directory
       0 [       0] address [size] of COR20 Header Directory
       0 [       0] address [size] of Reserved Directory


0:000> dds 60000000+19000 60000000+19000+2B0
60019064  7c82b0dc ntdll!RtlReAllocateHeap
60019068  77e4ec39 kernel32!HeapDestroy
6001906c  77e41fba kernel32!GetSystemTimeAsFileTime
60019070  77e619d1 kernel32!GetTickCount
60019074  77e69577 kernel32!QueryPerformanceCounter
60019078  7c82a9be ntdll!RtlSizeHeap
6001907c  77e82060 kernel32!SetUnhandledExceptionFilter
60019080  77e7690d kernel32!UnhandledExceptionFilter
60019084  77e42004 kernel32!TerminateProcess
60019088  7c82a136 ntdll!RtlRestoreLastWin32Error
6001908c  77e77a5f kernel32!SuspendThread
60019090  77e76a26 kernel32!SetThreadContext
60019094  77e77ae3 kernel32!GetThreadContext
60019098  77e73347 kernel32!FlushInstructionCache
6001909c  77e5f38b kernel32!ResumeThread
600190a0  77e616a8 kernel32!InterlockedCompareExchange
600190a4  77e645a9 kernel32!VirtualAlloc
600190a8  77e41fe3 kernel32!VirtualProtect
600190ac  77e66ed1 kernel32!VirtualQuery
600190b0  77e44960 kernel32!GetLogicalDriveStringsA
600190b4  77eab401 kernel32!GetVolumeNameForVolumeMountPointA
600190b8  77e6794d kernel32!GetACP
600190bc  77e6f3cf kernel32!GetLocaleInfoA
600190c0  77e622b7 kernel32!GetThreadLocale
600190c4  77e69d74 kernel32!GetVersionExA
600190c8  77e4beab kernel32!RaiseException
600190cc  77e60037 kernel32!GetSystemDirectoryA
600190d0  77e52bf4 kernel32!GetWindowsDirectoryA
600190d4  77e5c7a8 kernel32!lstrcmpA
600190d8  77e46c99 kernel32!OutputDebugStringA
600190dc  77e5bd7d kernel32!CreateEventA
600190e0  77e62311 kernel32!SetEvent
600190e4  77e51281 kernel32!ExpandEnvironmentStringsA
600190e8  77e9f365 kernel32!MoveFileA
600190ec  77e5da00 kernel32!IsDebuggerPresent
600190f0  77e9e4b1 kernel32!QueryDosDeviceA
600190f4  7c829e08 ntdll!RtlGetLastWin32Error
600190f8  77e63d7a kernel32!GetProcAddress
600190fc  77e41dc6 kernel32!LoadLibraryA
60019100  7c829e17 ntdll!RtlFreeHeap
60019104  77e62419 kernel32!LocalFree
60019108  7c829fd6 ntdll!RtlAllocateHeap
6001910c  77e63ec7 kernel32!GetProcessHeap
60019110  77ea2186 kernel32!VerifyVersionInfoA
60019114  7c81379f ntdll!VerSetConditionMask
60019118  77e63143 kernel32!WideCharToMultiByte
6001911c  77e70550 kernel32!SizeofResource
60019120  77e6b11b kernel32!SetHandleCount
60019124  77e69bf9 kernel32!LoadResource
60019128  77e511e1 kernel32!FindResourceA
6001912c  77e7388c kernel32!FindResourceExA
60019130  77e5be30 kernel32!lstrlenA
60019134  77e424de kernel32!Sleep
60019138  77ea2cb1 kernel32!WaitNamedPipeA
6001913c  77e63e6f kernel32!CloseHandle
60019140  77e5e123 kernel32!OpenEventA
60019144  77e622c9 kernel32!lstrlenW
60019148  77e62fc7 kernel32!GetCurrentThreadId
6001914c  77e5fdd4 kernel32!OpenProcess
60019150  77e63c78 kernel32!GetCurrentProcessId
60019154  7c81a3ab ntdll!RtlLeaveCriticalSection
60019158  7c81a360 ntdll!RtlEnterCriticalSection
6001915c  77e4cabf kernel32!GetComputerNameA
60019160  77e6f032 kernel32!ProcessIdToSessionId
60019164  77e645ff kernel32!GetModuleFileNameA
60019168  77e6474a kernel32!GetModuleHandleA
6001916c  77e62f9d kernel32!GetCurrentProcess
60019170  77e49968 kernel32!GetCurrentDirectoryA
60019174  77e61c7b kernel32!WaitForSingleObject
60019178  77e63868 kernel32!GetCurrentThread
6001917c  7c82c988 ntdll!RtlDeleteCriticalSection
60019180  77e67861 kernel32!InitializeCriticalSection
60019184  77e6b1a1 kernel32!FreeLibrary
60019188  77e63f41 kernel32!UnmapViewOfFile
6001918c  77e643f1 kernel32!MapViewOfFile
60019190  77e6b65f kernel32!OpenFileMappingA
60019194  77e61694 kernel32!InterlockedExchange
60019198  77e4d2fb kernel32!DeleteFileA
...
...
...
60019294  00000000
60019298  61e81000 detoured!Detoured
6001929c  00000000


This warrants the suspicion that 3rdPartyHook somehow optimized the virtual address space for its own purposes and this resulted in more fragmented virtual address space.

- Dmitry Vostokov @ DumpAnalysis.org -

7 Responses to “Crash Dump Analysis Patterns (Part 13e)”

  1. Crash Dump Analysis » Blog Archive » Hooksware Says:

    […] - Insufficient Memory (virtual memory) […]

  2. Crash Dump Analysis » Blog Archive » Insufficient Memory Patterns Says:

    […] Insufficient Memory (virtual memory) […]

  3. Crash Dump Analysis » Blog Archive » Invalid pointer, incorrect stack trace, multiple exceptions, insufficient memory and memory leak: pattern cooperation Says:

    […] of Applicatione.exe and get 2-3 subsequent process memory dumps, when it grows, to inspect its virtual memory growth using differential memory analysis […]

  4. Crash Dump Analysis » Blog Archive » Stack trace collection, message box, hidden exception, nested offender, insufficient memory, C++ exception, heap leak and ubiquitous component: pattern cooperation Says:

    […] box, find another hidden exception and from it see the real nested offender that experienced insufficient memory condition resulted in C++ exception. You can see WinDbg output from this dump in the post about […]

  5. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 26) Says:

    […] we introduce an icon for Insufficient Memory (virtual memory) […]

  6. Dmitry Vostokov Says:

    !address -summary WinDbg command also gives various output about region type usage and the largest available blocks per region type.

  7. Dmitry Vostokov Says:

    This pattern was renamed from “virtual memory” to “module fragmentation”

Leave a Reply