Software Diagnostics Library

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Advanced training sessions time may not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

• Title: Advanced Windows Memory Dump Analysis with Data Structures: Training Course Transcript and WinDbg Practice Exercises with Notes
• Description: The full transcript of Memory Dump Analysis Services Training with 10 step-by-step exercises, notes, and selected Q&A.
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Publisher: OpenTask (January 2012)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 180 pages
• ISBN-13: 978-1908043344

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As always, if I’m asked to do something, I don’t stop there and apply all my accumulated knowledge to go beyond. Here is an example: after designing ₂CARE² trademark I imagined an organic creature that catches bugs:

If you compare it with a trademark you would recognize A, R and E as Phenyl, Methyl, and Ethyl groups.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services started working on ₂CARE² system (Crash Analysis Report Environment, 2nd generation) and asked me to design a trademark. My Chemistry background (I like Organic Chemistry most) and imagination led me to represent client and server parts holistically as an aromatic-like compound:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having done in the past with user space raw stack data analysis for 32-bit complete memory dumps I found today the need to look at kernel raw stack data from all threads and created this fast script:

!for_each_thread "!thread @#Thread; r? $t1 = ((nt!_KTHREAD *) @#Thread )->StackLimit; r? $t2 = ((nt!_KTHREAD *) @#Thread )->InitialStack; dps @$t1 @$t2"

It can be run for kernel and complete memory dumps from both x86 and x64 systems. If you need to have correct symbolic mapping for user space in kernel space data you need to modify it a bit and it will be slower to run.

!for_each_thread "!thread @#Thread ff; .thread /r /p @#Thread; r? $t1 = ((nt!_KTHREAD *) @#Thread )->StackLimit; r? $t2 = ((nt!_KTHREAD *) @#Thread )->InitialStack; dps @$t1 @$t2"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I have discovered that Romantic era music is good for debugging, memory dump and software trace analysis sessions. Previously I included Beethoven and now suggest to listen to Schumann. For a starter you can try this album: Schumann: Symphonies Nos. 1-4

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

3 bugtations in a column:

Every debugger I know has trouble debugging.
Talent is helpful in debugging, but guts are absolutely necessary.
With failure comes a dump.

Joseph Heller

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here’s what we see from our Google Analytics stats. Top 5 browsers in 2011 used to access our portal and blog:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here’s what we see from our Google Analytics stats. Top 25 mobile devices in 2011 used to access our portal and blog:

Top 10 mobile operating systems:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

According to Google Analytics the number of visits / year increased by 4% since 2010 with almost 160,000 unique visitors (2% increase) from 180 countries and 34% of them are coming back. 2,725 visits were via 12 mobile operating systems (106% increase). Here are the top 100 network locations out of 28,932:

Top 25 visiting countries:

More than 8,000 portal and blog pages were viewed a total of more than 392,000 times with top 100 content pages:

More than 70,000 Google search keywords pointed to the portal and this blog with 100 most frequent (some are in Russian):

Special thanks to more  1,000 web sites that mention the portal and this blog with the first top 100:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This book I bought in a local Costa bookshop and found it was written by an Irish sociologist Kieran Allen. Shortly before my interest in Marxism was inspired by seeing a link to Irish communist party website and socialist bookshop in a booklet for Dublin Culture nights festival. It was a bit funny to see communists as part of Irish culture festival especially for me from former Soviet Union. Anyway, later I saw on streets that Marxist festivals are popular in Ireland nowadays. So let’s go back to the book. I found it very good and even lucid in explaining various Marxist ideas and vocabulary. A good start for more advance reading such as “Capital” (I have all 3 hardcover volumes from an Indian publisher and plan to have leather bound edition from Russia if I have enough surplus and MEW German edition) or specialized books such as “A Dictionary of Marxist Thought”. What I also tend to agree with the author is that Stalinism is a mirror of Capitalism (there is also a book “Magnetic Mountain: Stalinism as a Civilization” that I’m reading). I leave an application of a dialectical method of double negation to a reader here. Now the weak points of the book: 1) it doesn’t cover post-Stalinist era; 2) subsequent analysis of alternatives sounds a bit naive for me who really lived in socialism and can compare it to capitalism both in post-socialist country and now living in real capitalist country. The book also has a good reading suggestion list and I even thinking now on reading Voloshinov book “Marxism and the Philosophy of Language” (in Russian, although there is an English edition). Anyway, I would recommend Kieran’s book with reservations (about alternatives) as a first introduction to Marxist thought.

Marx and the Alternative to Capitalism

- Dmitry Vostokov @ LiterateScientist.com -

In addition to stack trace collections for threads (unmanaged, managed and predicate) we introduce an additional pattern for I/O requests. Such requests are implemented via the so called I/O request packets (IRP) that “travel” from a device driver to a device driver similar to a C++ class method to another C++ class method (where a device object address is similar to a C++ object instance address). An IRP stack is used to keep a track of the current driver which is processing an IRP that is reused between device drivers. Its is basically an array of structures describing how a particular driver function was called with appropriate parameters similar to a call frame on an execution thread stack. Long time ago I created an UML diagram depicting the flow of an IRP through the driver (device) stack (diagram #3). An I/O stack location pointer is decremented (from the bottom to the top) like a thread stack pointer (ESP or RSP). We can list active and completed I/O requests with their stack traces using !irpfind -v WinDbg command:

1: kd> !irpfind -v

Scanning large pool allocation table for Tag: Irp? (832c7000 : 833c7000)

Irp    [ Thread ] irpStack: (Mj,Mn)   DevObj  [Driver]         MDL Process
8883dc18: Irp is active with 1 stacks 1 is current (= 0x8883dc88)
No Mdl: No System Buffer: Thread 888f8950:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  d, 0]   5  1 88515ae8 888f82f0 00000000-00000000    pending
\FileSystem\Npfs
Args: 00000000 00000000 00110008 00000000

891204c8: Irp is active with 1 stacks 1 is current (= 0x89120538)
No Mdl: No System Buffer: Thread 889635b0:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 84752028 00000000-00000000    pending
\FileSystem\Npfs
Args: 0000022a 00000000 00000000 00000000

89120ce8: Irp is active with 1 stacks 1 is current (= 0x89120d58)
No Mdl: No System Buffer: Thread 89212030:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 8921be00 00000000-00000000    pending
\FileSystem\Npfs
Args: 0000022a 00000000 00000000 00000000
Searching NonPaged pool (80000000 : ffc00000) for Tag: Irp?

[...]

892cbe48: Irp is active with 9 stacks 9 is current (= 0x892cbfd8)
No Mdl: No System Buffer: Thread 892add78:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
>[  c, 2]   0  1 8474a020 892c8c80 00000000-00000000    pending
\FileSystem\Ntfs
Args: 00000800 00000002 00000000 00000000

892daa88: Irp is active with 4 stacks 4 is current (= 0x892dab64)
No Mdl: System buffer=831559c8: Thread 8322c8e8:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
>[  e,2d]   5  1 884ba750 83190c40 00000000-00000000    pending
\Driver\AFD
Args: 890cbc44 890cbc44 88e55297 8943b6c8

892ea4e8: Irp is active with 4 stacks 4 is current (= 0x892ea5c4)
No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  Pending has been returned
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  2 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 c0000185
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  f, 0]   0  2 83a34bb0 00000000 84d779ed-88958050
\Driver\atapi CLASSPNP!ClasspMediaChangeDetectionCompletion
Args: 88958050 00000000 00000000 83992d10
>[  0, 0]   2  0 891ee030 00000000 00000000-00000000
\Driver\cdrom
Args: 00000000 00000000 00000000 00000000

8933fcb0: Irp is active with 1 stacks 1 is current (= 0x8933fd20)
No Mdl: No System Buffer: Thread 84753d78:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 84759f40 00000000-00000000    pending
\FileSystem\Npfs
Args: 0000022a 00000000 00000000 00000000

893cf550: Irp is active with 1 stacks 1 is current (= 0x893cf5c0)
No Mdl: No System Buffer: Thread 888fd3b8:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 834d30d0 00000000-00000000    pending
\FileSystem\Npfs
Args: 00000400 00000000 00000000 00000000

893da468: Irp is active with 6 stacks 7 is current (= 0x893da5b0)
Mdl=892878f0: No System Buffer: Thread 00000000:  Irp is completed.  Pending has been returned
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  f, 0]   0  0 84b3e028 00000000 9747fcd0-00000000
\Driver\usbehci USBSTOR!USBSTOR_CswCompletion
Args: 00000000 00000000 00000000 00000000
[  f, 0]   0  0 892ba8f8 00000000 84d780ce-8328e0f0
\Driver\USBSTOR CLASSPNP!TransferPktComplete
Args: 00000000 00000000 00000000 00000000

893efb00: Irp is active with 10 stacks 11 is current (= 0x893efcd8)
Mdl=83159378: No System Buffer: Thread 82b7f828:  Irp is completed.  Pending has been returned
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 885a55b8 00000000 81614138-00000000
\Driver\disk partmgr!PmReadWriteCompletion
Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 89257c90 00000000 8042e4d4-831caab0
\Driver\partmgr volmgr!VmpReadWriteCompletionRoutine
Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 831ca9f8 00000000 84dad0be-00000000
\Driver\volmgr ecache!EcDispatchReadWriteCompletion
Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 8319c020 00000000 84dcc4d4-8576f8ac
\Driver\Ecache volsnap!VspSignalCompletion
Args: 00000000 00000000 00000000 00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

An advice to succeed in Software Technical Support:

They said : The dump came from the wrong site
He’s a page from the end of the dump
I said : The dump came from the right site
Took me just one look
<…>
With a little debugging
You will survive
With a little debugging
You will get by
Do what you want
Go your own way
in Technical Support.

Modern Talking

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is another tracing example of unified debugging patterns introduced previously.

- Analysis Patterns

Focus of Tracing

- Architectural Patterns

Debug Event Subscription / Notification

- Design Patterns

Shared Debug Event State

- Implementation Patterns

Shared Variable

- Usage Patterns

Saving a memory address obtained at a breakpoint event in a debugger pseudo-register for use at later breakpoint events

Debugging.tv published a case study for tracing window messages in WinDbg. There a pseudo-register is used to save a buffer address before GetMessage call and then to reuse it after the call. Please look at Event State Management slide on Frames episode 0×06 presentation. The full WinDbg log and the recording are available there too.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a seasonal greetings postcard with Windows 8 bluescreen theme:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Based on ideas of Roman Jakobson about “marked” and “unmarked” categories we propose another pattern called Marked Message that groups trace messages based on having some feature or property. For example, marked messages may point to some domain of software activity such related to functional requirements and therefore may help in troubleshooting and debugging. Unmarked messages include all other messages that don’t say anything about such activities (although may include messages pointing to such activities indirectly we unaware of) or messages that say explicitly that no such activity has occurred. We can even borrow a notation of distinctive features from phonology and annotate any trace or log after analysis to compare it with a master trace, for example, compose the following list of software trace distinctive features:

session database queries [+]
session initialization [-]
socket activity [+]
process A launched [+]
process B launched [-]
process A exited [-]
[…]

Here [+] means the activity is present in the trace and [-]  means the activity is either undetected or definitely not present. Sometime a non-present activity can be a marked activity corresponding to all inclusive unmarked present activity (see, for example, No Activity pattern).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A WinDbg snippet from a multithreaded service:

0:2011> ~2012s
0:2012>

PS. Teaching WinDbg commands on the eve!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The beginning of a debugging tale:

He was debugging for 7 years and when he stopped and looked around he saw only 2 people left from the team of 50 strong technical support engineers.

Folktale

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have memory leaks related to the growing number of page tables. One reason for that could be the growing number of zombie processes (noticeable with tens of thousands of them).

1: kd> !process 0 0
[...] 
PROCESS fffffa80266bd6f0
    SessionId: 0  Cid: 0a6c    Peb: 7fffffdc000  ParentCid: 03ac
    DirBase: 9d35a000  ObjectTable: fffff8a00170ac80  HandleCount: 152.
    Image: svchost.exe
[…] 
PROCESS fffffa8027de9b30
    SessionId: 0  Cid: 21d0    Peb: 7fffffdf000  ParentCid: 02e0
    DirBase: 37881000  ObjectTable: 00000000  HandleCount:   0.
    Image: conhost.exe
[…] 
PROCESS fffffa8028eb0600
    SessionId: 0  Cid: ab88    Peb: 7fffffdf000  ParentCid: 02e0
    DirBase: 27a2f000  ObjectTable: 00000000  HandleCount:   0.
    Image: conhost.exe
[…]

Even zombies have at least one remaining page (page directory) from the former page tables of their virtual to physical memory mapping (!dd is the same as dd command but for physical memory):

1: kd> !dd 9d35a000
#9d35a000 9dd62867 03c00000 00000000 00000000
#9d35a010 00000000 00000000 00000000 00000000
#9d35a020 00000000 00000000 00000000 00000000
#9d35a030 00000000 00000000 00000000 00000000
#9d35a040 00000000 00000000 00000000 00000000
#9d35a050 00000000 00000000 00000000 00000000
#9d35a060 00000000 00000000 00000000 00000000
#9d35a070 00000000 00000000 9d45e867 49500000

1: kd> !dd 37881000
#37881000 00000000 00000000 00000000 00000000
#37881010 00000000 00000000 00000000 00000000
#37881020 00000000 00000000 00000000 00000000
#37881030 00000000 00000000 00000000 00000000
#37881040 00000000 00000000 00000000 00000000
#37881050 00000000 00000000 00000000 00000000
#37881060 00000000 00000000 00000000 00000000
#37881070 00000000 00000000 00000000 00000000

1: kd> !dd 27a2f000
#27a2f000 00000000 00000000 00000000 00000000
#27a2f010 00000000 00000000 00000000 00000000
#27a2f020 00000000 00000000 00000000 00000000
#27a2f030 00000000 00000000 00000000 00000000
#27a2f040 00000000 00000000 00000000 00000000
#27a2f050 00000000 00000000 00000000 00000000
#27a2f060 00000000 00000000 00000000 00000000
#27a2f070 00000000 00000000 00000000 00000000

We also see that 2 conhost.exe processes have identical physical to virtual mapping because their user space mappings are no longer valid (zeroed) and svchost.exe process has user space mapping (in blue italics):

1: kd> !ptov 27a2f000
Amd64PtoV: pagedir 27a2f000
27a2f000 fffff6fb`7dbed000
71530000 fffff6fb`7dbee000
19d000 fffff6fb`7dbef000
199000 fffff6fb`7dbf0000
b6a04000 fffff6fb`7dbf1000
b1f57000 fffff6fb`7dbf2000
29c4000 fffff6fb`7dbf3000
1c53000 fffff6fb`7dbf5000 
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

1: kd> !ptov 37881000
Amd64PtoV: pagedir 37881000
37881000 fffff6fb`7dbed000
8d482000 fffff6fb`7dbee000
19d000 fffff6fb`7dbef000
199000 fffff6fb`7dbf0000
b6a04000 fffff6fb`7dbf1000
b1f57000 fffff6fb`7dbf2000
29c4000 fffff6fb`7dbf3000
1c53000 fffff6fb`7dbf5000
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

1: kd> !ptov 9d35a000
Amd64PtoV: pagedir 9d35a000
9e587000 10000
6871e000 20000
af5aa000 30000
af5ab000 31000
afaac000 32000
afbad000 33000
af2f5000 40000
9d66b000 50000
22199000 60000
9d962000 e5000
9d261000 e6000
9dc60000 e7000
9d256000 ea000
9d84f000 eb000
9e4ec000 ec000
9e081000 ed000
9d876000 ee000
9e271000 ef000
b8bfd000 f0000
b8efe000 f1000
b86ff000 f2000
b5302000 f3000
b5202000 f4000
b5502000 f5000
b7f03000 f6000
b8404000 f7000
b8415000 100000
b8b16000 101000
b1b17000 102000
[…]
2cd4000 77512000
5d7000 77515000
5d8000 77516000
4d9000 77517000
b358f000 77590000
aef04000 77591000
68624000 77592000
64b26000 77593000
af4c6000 77595000
b2042000 7efe0000
b2143000 7efe1000
b1a56000 7efe2000
b1a57000 7efe3000
b1b58000 7efe4000
1ba000 7ffe0000
9da69000 bfeb0000
aeeae000 ffea0000
af191000 ffea1000
9d76a000 ffea2000
ae793000 ffea3000
9dc8e000 ffea5000
b7eb7000 ffea6000
9dffc000 ffea7000
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

In order to check user space virtual addresses we have to switch to the corresponding process context:

1: kd> !pte fffffa80`28f2d000
                                           VA fffffa8028f2d000
PXE at FFFFF6FB7DBEDFA8    PPE at FFFFF6FB7DBF5000    PDE at FFFFF6FB7EA00A38    PTE at FFFFF6FD40147968
contains 0000000001C53863  contains 0000000001C54863  contains 0000000049320863  contains 000000002E4D8963
pfn 1c53      —DA–KWEV  pfn 1c54      —DA–KWEV  pfn 49320     —DA–KWEV  pfn 2e4d8     -G-DA–KWEV

1: kd> .process /r /p fffffa80266bd6f0
Implicit process is now fffffa80`266bd6f0
Loading User Symbols

1: kd> !pte 10000
                                           VA 0000000000010000
PXE at FFFFF6FB7DBED000    PPE at FFFFF6FB7DA00000    PDE at FFFFF6FB40000000    PTE at FFFFF68000000080
contains 03C000009DD62867  contains 031000009D865867  contains 7C2000009DD66867  contains 9CB000009E587867
pfn 9dd62     —DA–UWEV  pfn 9d865     —DA–UWEV  pfn 9dd66     —DA–UWEV  pfn 9e587     —DA–UW-V

This pattern came to our attention after several customers complained about the growing number of memory allocated for page tables which exceeded a gigabyte after several days.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -