Crash Dump Analysis Patterns (Part 13d)
In order to maintain virtual to physical address translation OS needs page tables. These tables occupy memory too. If there is not enough memory for new tables the system will fail to create processes, allocate I/O buffers and memory from pools. You might see the following diagnostic message from WinDbg:
4: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 851422 ( 3405688 Kb)
Page File: \??\C:\pagefile.sys
Current: 2095104 Kb Free Space: 2081452 Kb
Minimum: 2095104 Kb Maximum: 4190208 Kb
Available Pages: 683464 ( 2733856 Kb)
ResAvail Pages: 800927 ( 3203708 Kb)
Locked IO Pages: 145 ( 580 Kb)
Free System PTEs: 23980 ( 95920 Kb)
******* 356363 system PTE allocations have failed ******
Free NP PTEs: 6238 ( 24952 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 482 ( 1928 Kb)
Modified PF Pages: 482 ( 1928 Kb)
NonPagedPool Usage: 18509 ( 74036 Kb)
NonPagedPool Max: 31970 ( 127880 Kb)
PagedPool 0 Usage: 8091 ( 32364 Kb)
PagedPool 1 Usage: 2495 ( 9980 Kb)
PagedPool 2 Usage: 2580 ( 10320 Kb)
PagedPool 3 Usage: 2552 ( 10208 Kb)
PagedPool 4 Usage: 2584 ( 10336 Kb)
PagedPool Usage: 18302 ( 73208 Kb)
PagedPool Maximum: 39936 ( 159744 Kb)
********** 48530 pool allocations have failed **********
Shared Commit: 5422 ( 21688 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 5762 ( 23048 Kb)
PagedPool Commit: 18365 ( 73460 Kb)
Driver Commit: 2347 ( 9388 Kb)
Committed pages: 129014 ( 516056 Kb)
Commit limit: 1342979 ( 5371916 Kb)
We also see another diagnostic message about pool allocation failures which could be the consequence of PTE allocation failures.
The cause of system PTE allocation failures might be incorrect value of SystemPages registry key that needs to be adjusted as explained in the following TechNet article:
The number of free page table entries is low, which can cause system instability
Another cause would be /3GB boot option on x86 systems especially used for hosting terminal sessions. This case is explained in Brad Rutkowski’s blog post which also shows how to detect /3GB kernel and complete memory dumps:
Consequences of running 3GB and PAE together
In our case the system was booted with /3GB:
4: kd> vertarget
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer
Built by: 3790.srv03_sp2_gdr.070304-2240
Kernel base = 0xe0800000 PsLoadedModuleList = 0xe08af9c8
Debug session time: Fri Feb 1 09:10:17.703 2008 (GMT+0)
System Uptime: 6 days 17:14:45.528
Normal Windows 2003 systems have different kernel base address which can be checked from Reference Stack Traces for Windows Server 2003 (Virtual Memory section):
kd> vertarget
Windows Server 2003 Kernel Version 3790 (Service Pack 2) UP Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 3790.srv03_sp2_rtm.070216-1710
Kernel base = 0×80800000 PsLoadedModuleList = 0×8089ffa8
Debug session time: Wed Jan 30 17:54:13.390 2008 (GMT+0)
System Uptime: 0 days 0:30:12.000
- Dmitry Vostokov @ DumpAnalysis.org -
February 21st, 2008 at 3:03 am
You shoul see my new post on SYSPTES, they just fixed it so you can sue public symbols again:
http://blogs.technet.com/brad_rutkowski/archive/2008/02/21/i-pte-the-fool-sysptes-4-works-in-vista-sp1-ws08.aspx
July 28th, 2009 at 5:38 pm
[…] increaseuserva in BCD (p. 14) - this might get your terminal server installation into problems. See example of Insufficient Memory (PTE) pattern. […]
April 12th, 2010 at 10:05 am
Dmitry,
The following is a live KD, in VM we are observed 8000+ cmd.exe instances running on the context of system.
My Question is how can an system uptime is 22 days and KernelTime is 700+ Days for that processor ?
lkd> .time
Debug session time: Mon Apr 12 06:07:15.797 2010 (GMT-4)
System Uptime: 22 days 10:41:32.579
lkd> !process 00a4
Searching for Process with Cid == a4
Cid Handle table at e29a6000 with 10925 Entries in use
PROCESS fb32cd88 SessionId: 0 Cid: 00a4 Peb: 7ffff000 ParentCid: 2ce0
DirBase: bff54960 ObjectTable: e3c046a0 HandleCount: 0.
Image: cmd.exe
VadRoot fcc99398 Vads 6 Clone 0 Private 4. Modified 0. Locked 0.
DeviceMap e16008e8
Token e3ce6030
ElapsedTime 21 Days 14:20:04.071
UserTime 00:00:00.000
KernelTime 762 Days 03:42:46.250
QuotaPoolUsage[PagedPool] 6780
QuotaPoolUsage[NonPagedPool] 160
Working Set Sizes (now,min,max) (16, 50, 345) (64KB, 200KB, 1380KB)
PeakWorkingSetSize 16
VirtualSize 1 Mb
PeakVirtualSize 1 Mb
PageFaultCount 9
MemoryPriority BACKGROUND
BasePriority 10
CommitCharge 39
No active threads
April 14th, 2010 at 6:56 pm
I find this in many dumps and have a suspicion it is related to “orphaned” processes:
HandleCount: 0
No active threads
Thanks,
Dmitry
April 16th, 2010 at 2:13 pm
[…] we introduce an icon for Insufficient Memory (PTE) […]
March 25th, 2016 at 9:45 pm
According to Windows Internals we can set HKLM\S\CCS\C\Session Manager\TrackPtes key value to 1 and then see allocation history by using !sysptes 4 WinDbg command.