Sparse complete x64 memory dumps
Forthcoming Webinar on Pattern-Driven Software Diagnostics
2012 - The Year of Software Trace Analysis
Accelerated Memory Dump Analysis Training
Sponsored link: Memory Dump Analysis Services
Debugging Experts Magazine Online
Debugging Today Daily Newspaper
Because of the larger virtual address space x64 Windows servers are usually equipped with 16Gb or more physical memory to take advantage of new vast memory layout where pools are “virtually” unlimited and their size is measured in Gb than in Mb (highlighted in enlarged blue font below):
0: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 4193970 ( 16775880 Kb)
Page File: \??\C:\pagefile.sys
Current: 17825792 Kb Free Space: 17810140 Kb
Minimum: 17825792 Kb Maximum: 17825792 Kb
Page File: \??\D:\pagefile.sys
Current: 32768000 Kb Free Space: 32754984 Kb
Minimum: 32768000 Kb Maximum: 32768000 Kb
Available Pages: 3851036 ( 15404144 Kb)
ResAvail Pages: 3951755 ( 15807020 Kb)
Locked IO Pages: 136 ( 544 Kb)
Free System PTEs: 16752738 ( 67010952 Kb)
Free NP PTEs: 1635326 ( 6541304 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 52 ( 208 Kb)
Modified PF Pages: 38 ( 152 Kb)
NonPagedPool Usage: 12421 ( 49684 Kb)
NonPagedPool Max: 1668607 ( 6674428 Kb)
PagedPool 0 Usage: 9501 ( 38004 Kb)
PagedPool 1 Usage: 604 ( 2416 Kb)
PagedPool 2 Usage: 616 ( 2464 Kb)
PagedPool 3 Usage: 598 ( 2392 Kb)
PagedPool 4 Usage: 603 ( 2412 Kb)
PagedPool Usage: 11922 ( 47688 Kb)
PagedPool Maximum: 6674432 ( 26697728 Kb)
Shared Commit: 2649 ( 10596 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 8472 ( 33888 Kb)
PagedPool Commit: 11949 ( 47796 Kb)
Driver Commit: 2603 ( 10412 Kb)
Committed pages: 159687 ( 638748 Kb)
Commit limit: 16686113 ( 66744452 Kb)
[...]
It came to my attention today that complete memory dumps can be smaller, sparser in such big memory layouts with many unused physical memory regions. Therefore, complete memory dumps could be smaller than the actual amount of physical memory and even when possibly truncated with many OS structures being included. For the virtual memory stats above the size of complete memory dump was 5Gb and although WinDbg reports the dump as truncated with 16Gb of physical memory it was possible that everything was fit into the first 5Gb of physical memory and saved accordingly in 17Gb page file. For example, !locks command works perfectly (it frequently unable to traverse truncated complete dumps from 32-bit Windows):
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...
Resource @ nt!CmpRegistryLock (0xfffff800011de220) Shared 1 owning threads
Contention Count = 11
Threads: fffffade708e17a0-01<*>
KD: Scanning for held locks...
Resource @ 0xfffffade6f8b1a40 Shared 1 owning threads
Threads: fffffade708e17a0-01<*>
KD: Scanning for held locks...
6213 total locks, 2 locks currently held
At the same time some data is missing from the file so it could be really truncated dump. For example, the information about computer name is missing:
0: kd> dq srv!srvcomputername l2
fffffade`57919a10 00000000`00220010 fffffa80`01cfa980
0: kd> !address fffffade`57919a10
fffffade55e04000 - 0000000005ffb000 ffade6e1108e0
Usage KernelSpaceUsageNonPagedSystem
0: kd> !pte fffffade`57919a10
VA fffffade57919a10
PXE @ FFFFF6FB7DBEDFA8 PPE at FFFFF6FB7DBF5BC8 PDE at FFFFF6FB7EB795E0 PTE at FFFFF6FD6F2BC8C8
contains 0000000114E00863 contains 000000011CD63863 contains 000000011CE20963 contains 80000000A8265963
pfn 114e00 —DA–KWEV pfn 11cd63 —DA–KWEV pfn 11ce20 -G-DA–KWEV pfn a8265 -G-DA–KW-V
0: kd> du fffffa80`01cfa980 l10
fffffa80`01cfa980 “????????????????”
0: kd> !address fffffa80`01cfa980
fffffa8000000000 - 000000065d800000 ffade6e1108e0
Usage KernelSpaceUsagePagedPool
0: kd> !pte fffffa80`01cfa980
VA fffffa8001cfa980
PXE @ FFFFF6FB7DBEDFA8 PPE at FFFFF6FB7DBF5000 PDE at FFFFF6FB7EA00070 PTE at FFFFF6FD4000E7D0
Unable to get PDE FFFFF6FB7EA00070
Fortunately I got the computer name from a PEB of a randomly selected process though:
0: kd> .process /r /p fffffade6ddd9c20
Implicit process is now fffffade`6ddd9c20
Loading User Symbols
...
0: kd> !peb
PEB at 000000007efdf000
[...]
COMPUTERNAME=SERVER_A
[…]
I remember that during my Florida trip almost 5 years ago people were worrying about troubleshooting crashes and hangs on 64-bit Windows and discussed how they would send zipped complete memory dumps on several DVD via a courier post. Now with Blu-ray discs (BD) becoming a commodity the size of complete memory dumps is not perceived as a big problem… For really huge dumps WinDbg scripts collecting data on-site might be a solution too (see Dmp2Txt: Solving Security Problem for WinDbg script usage).
- Dmitry Vostokov @ DumpAnalysis.org -
Sponsored link: Professional Software Debugging Services
/* Malware and Software Defects -> Victimware.org */
Copyright © 2006 - 2012. This is a non-profit research and scientific project.
_1125.png)
Citrix and Microsoft Customer Forum
Museum of Debugging and Memory Dumps
7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story
Crash and Hang Analysis Audit Service
CARE: Crash Analysis Report Environment
Crash Dump and Software Trace Analysis Training and Seminars
Access OpenTask Titles on Safari Books Online
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
International Memory Analysts and Debuggers Day: 07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA
AnnouncementsComing Soon:
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Management Bits: An Anthology from Reductionist Manager
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Introduction to Pattern-Driven Software Problem Solving
Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5
Windows Debugging Notebook: Essential User Space WinDbg Commands
Memory Dump Analysis Anthology, Volume 5
Memory Dump Analysis Anthology, Volume 4
Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
November 2nd, 2008 at 9:15 am
!memusage should give physical memory stats