Software Diagnostics Library

Because of the larger virtual address space x64 Windows servers are usually equipped with 16Gb or more physical memory to take advantage of new vast memory layout where pools are “virtually” unlimited and their size is measured in Gb than in Mb (highlighted in enlarged blue font below):

0: kd> !vm

*** Virtual Memory Usage ***
        Physical Memory:     4193970 (  16775880 Kb)
        Page File: \??\C:\pagefile.sys
          Current:  17825792 Kb  Free Space:  17810140 Kb
          Minimum:  17825792 Kb  Maximum:     17825792 Kb
        Page File: \??\D:\pagefile.sys
          Current:  32768000 Kb  Free Space:  32754984 Kb
          Minimum:  32768000 Kb  Maximum:     32768000 Kb
        Available Pages:     3851036 (  15404144 Kb)
        ResAvail Pages:      3951755 (  15807020 Kb)
        Locked IO Pages:         136 (       544 Kb)
        Free System PTEs:   16752738 (  67010952 Kb)
        Free NP PTEs:        1635326 (   6541304 Kb)
        Free Special NP:           0 (         0 Kb)
        Modified Pages:           52 (       208 Kb)
        Modified PF Pages:        38 (       152 Kb)
        NonPagedPool Usage:    12421 (     49684 Kb)
        NonPagedPool Max:    1668607 (   6674428 Kb)
        PagedPool 0 Usage:      9501 (     38004 Kb)
        PagedPool 1 Usage:       604 (      2416 Kb)
        PagedPool 2 Usage:       616 (      2464 Kb)
        PagedPool 3 Usage:       598 (      2392 Kb)
        PagedPool 4 Usage:       603 (      2412 Kb)
        PagedPool Usage:       11922 (     47688 Kb)
        PagedPool Maximum:   6674432 (  26697728 Kb)
        Shared Commit:          2649 (     10596 Kb)
        Special Pool:              0 (         0 Kb)
        Shared Process:         8472 (     33888 Kb)
        PagedPool Commit:      11949 (     47796 Kb)
        Driver Commit:          2603 (     10412 Kb)
        Committed pages:      159687 (    638748 Kb)
        Commit limit:       16686113 (  66744452 Kb)

[...]

It came to my attention today that complete memory dumps can be smaller, sparser in such big memory layouts with many unused physical memory regions. Therefore, complete memory dumps could be smaller than the actual amount of physical memory and even when possibly truncated with many OS structures being included. For the virtual memory stats above the size of complete memory dump was 5Gb and although WinDbg reports the dump as truncated with 16Gb of physical memory it was possible that everything was fit into the first 5Gb of physical memory and saved accordingly in 17Gb page file. For example, !locks command works perfectly (it frequently unable to traverse truncated complete dumps from 32-bit Windows):

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...

Resource @ nt!CmpRegistryLock (0xfffff800011de220)    Shared 1 owning threads
    Contention Count = 11
     Threads: fffffade708e17a0-01<*>
KD: Scanning for held locks...

Resource @ 0xfffffade6f8b1a40    Shared 1 owning threads
     Threads: fffffade708e17a0-01<*>
KD: Scanning for held locks...

6213 total locks, 2 locks currently held

At the same time some data is missing from the file so it could be really truncated dump. For example, the information about computer name is missing:

0: kd> dq srv!srvcomputername l2
fffffade`57919a10  00000000`00220010 fffffa80`01cfa980

0: kd> !address fffffade`57919a10
  fffffade55e04000 - 0000000005ffb000           ffade6e1108e0
          Usage       KernelSpaceUsageNonPagedSystem

0: kd> !pte fffffade`57919a10
                                 VA fffffade57919a10
PXE @ FFFFF6FB7DBEDFA8     PPE at FFFFF6FB7DBF5BC8    PDE at FFFFF6FB7EB795E0    PTE at FFFFF6FD6F2BC8C8
contains 0000000114E00863  contains 000000011CD63863  contains 000000011CE20963  contains 80000000A8265963
pfn 114e00 —DA–KWEV        pfn 11cd63 —DA–KWEV        pfn 11ce20 -G-DA–KWEV        pfn a8265 -G-DA–KW-V

0: kd> du fffffa80`01cfa980 l10
fffffa80`01cfa980  “????????????????”

0: kd> !address fffffa80`01cfa980
  fffffa8000000000 - 000000065d800000           ffade6e1108e0
          Usage       KernelSpaceUsagePagedPool

0: kd> !pte fffffa80`01cfa980
                                 VA fffffa8001cfa980
PXE @ FFFFF6FB7DBEDFA8     PPE at FFFFF6FB7DBF5000    PDE at FFFFF6FB7EA00070    PTE at FFFFF6FD4000E7D0
Unable to get PDE FFFFF6FB7EA00070

Fortunately I got the computer name from a PEB of a randomly selected process though:

0: kd> .process /r /p fffffade6ddd9c20
Implicit process is now fffffade`6ddd9c20
Loading User Symbols
...

0: kd> !peb
PEB at 000000007efdf000
[...]
        COMPUTERNAME=SERVER_A
[…]

I remember that during my Florida trip almost 5 years ago people were worrying about troubleshooting crashes and hangs on 64-bit Windows and discussed how they would send zipped complete memory dumps on several DVD via a courier post. Now with Blu-ray discs (BD) becoming a commodity the size of complete memory dumps is not perceived as a big problem… For really huge dumps WinDbg scripts collecting data on-site might be a solution too (see Dmp2Txt: Solving Security Problem for WinDbg script usage).

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Thursday, October 30th, 2008 at 4:49 pm and is filed under Crash Dump Analysis, WinDbg Scripts. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.