Software Diagnostics Library

Although we briefly mentioned session pool in Insufficient Memory (kernel pool) pattern we decided to factor it into a separate (sub)pattern and provide WinDbg commands to analyze possible leaks. The following output shows the sequence of commands that gives you an idea although the example itself was taken from a healthy dump so no red coloring (from my memory leaks in session pool happened mostly in 32-bit past):

1: kd> !vm 4

Terminal Server Memory Usage By Session:

Session ID 0 @ fffff8800324d000:
Paged Pool Usage:        4128K
Commit Usage:            7488K

Session ID 1 @ fffff88002f65000:
Paged Pool Usage:       32852K
Commit Usage:           36488K

1: kd> !session
Sessions on machine: 2
Valid Sessions: 0 1
Error in reading current session

1: kd> !session -s 1
Sessions on machine: 2
Implicit process is now fffffa80`07d79730
Using session 1

1: kd> !poolused 8
Sorting by Session Tag

Pool Used:
NonPaged            Paged
Tag    Allocs     Used    Allocs     Used
TOTAL           4     4208      9500 33475120
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As was announced earlier we start cataloguing elemental malware detection and analysis patterns. We skip Part 1 because we assign Deviant Module to it. Part 2 deals with Fake Module pattern where one of loaded modules masquerades as a legitimate system DLL or a widely known value adding DLL from some popular 3rd party product. To illustrate this pattern we modeled it as Victimware: a process crashed after loading a malware module:

0:000> k
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
00000000`0026f978 00000001`3f89103a 0x0
00000000`0026f980 00000001`3f8911c4 FakeModule!wmain+0x3a
00000000`0026f9c0 00000000`76e3652d FakeModule!__tmainCRTStartup+0x144
00000000`0026fa00 00000000`7752c521 kernel32!BaseThreadInitThunk+0xd
00000000`0026fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

When we inspect loaded modules we don’t find anything suspicious:

0:000> lmp
start             end                 module name
00000000`76e20000 00000000`76f3f000   kernel32 <none>
00000000`77500000 00000000`776a9000   ntdll    <none>
00000001`3f890000 00000001`3f8a6000   FakeModule <none>
000007fe`f8cb0000 000007fe`f8cc7000   winspool <none>
000007fe`fdb30000 000007fe`fdb9c000   KERNELBASE <none>

However, when checking modules images for any modifications we find that winspool was not compared with existing binary from Microsoft symbol server:

0:000> !for_each_module "!chkimg -v -d @#ModuleName"
Searching for module with expression: kernel32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: C:\WSDK8\Debuggers\x64\sym\kernel32.dll\503285C111f000\kernel32.dll
No range specified

Scanning section:    .text
Size: 633485
Range to scan: 76e21000-76ebba8d
Total bytes compared: 633485(100%)
Number of errors: 0
0 errors : kernel32
Searching for module with expression: ntdll
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: C:\WSDK8\Debuggers\x64\sym\ntdll.dll\4EC4AA8E1a9000\ntdll.dll
No range specified

Scanning section:    .text
Size: 1049210
Range to scan: 77501000-7760127a
Total bytes compared: 1049210(100%)
Number of errors: 0

Scanning section:       RT
Size: 474
Range to scan: 77602000-776021da
Total bytes compared: 474(100%)
Number of errors: 0
0 errors : ntdll
Searching for module with expression: FakeModule
Error for FakeModule: Could not find image file for the module. Make sure binaries are included in the symbol path.
Searching for module with expression: winspool
Error for winspool: Could not find image file for the module. Make sure binaries are included in the symbol path.
Searching for module with expression: KERNELBASE
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: C:\WSDK8\Debuggers\x64\sym\KERNELBASE.dll\503285C26c000\KERNELBASE.dll
No range specified

Scanning section:    .text
Size: 302047
Range to scan: 7fefdb31000-7fefdb7abdf
Total bytes compared: 302047(100%)
Number of errors: 0
0 errors : KERNELBASE

Checking module data reveals that it was loaded not from System32 folder and doesn’t have any version information:

0:000> lmv m winspool
start             end                 module name
000007fe`f8cb0000 000007fe`f8cc7000   winspool   (deferred)
Image path: C:\Work\AWMA\FakeModule\x64\Release\winspool.drv
Image name: winspool.drv
Timestamp:        Fri Dec 28 22:22:42 2012 (50DE1BB2)
CheckSum:         00000000
ImageSize:        00017000
File version:     0.0.0.0
Product version:  0.0.0.0
File flags:       0 (Mask 0)
File OS:          0 Unknown Base
File type:        0.0 Unknown
File date:        00000000.00000000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

We could see that path from running this command as well :

0:000> !for_each_module
00: 0000000076e20000  0000000076f3f000         kernel32 C:\Windows\System32\kernel32.dll                      kernel32.dll
01: 0000000077500000  00000000776a9000            ntdll C:\Windows\System32\ntdll.dll                            ntdll.dll
02: 000000013f890000  000000013f8a6000       FakeModule C:\Work\AWMA\FakeModule\x64\Release\FakeModule.exe  FakeModule.exe
03: 000007fef8cb0000  000007fef8cc7000         winspool C:\Work\AWMA\FakeModule\x64\Release\winspool.drv
04: 000007fefdb30000  000007fefdb9c000       KERNELBASE C:\Windows\System32\KERNELBASE.dll                  KERNELBASE.dll

or from PEB:

0:000> !peb
PEB at 000007fffffdf000
[...]
7fef8cb0000 50de1bb2 Dec 28 22:22:42 2012 C:\Work\AWMA\FakeModule\x64\Release\winspool.drv
[…]

Another sign is module size in memory which is much smaller than real winspool.drv:

0:000> ? 000007fe`f8cc7000 - 000007fe`f8cb0000
Evaluate expression: 94208 = 00000000`0001700

Module size can help if legitimate module from well-known folder was replaced. Module debug directory and the size of export and import directories are also different with the former revealing the development folder:

0:000> !dh 000007fe`f8cb0000
[...]
   0 [       0] address [size] of Export Directory
[…]
9000 [     208] address [size] of Import Address Table Directory
[…]
Debug Directories(2)
Type       Size     Address  Pointer
cv           49        e2c0     cac0 Format: RSDS, guid, 1, C:\Work\AWMA\FakeModule\x64\Release\winspool.pdb

This can also be seen from the output of !lmi command:

0:000> !lmi 7fef8cb0000
Loaded Module Info: [7fef8cb0000]
Module: winspool
Base Address: 000007fef8cb0000
Image Name: winspool.drv
Machine Type: 34404 (X64)
Time Stamp: 50de1bb2 Fri Dec 28 22:22:42 2012
Size: 17000
CheckSum: 0
Characteristics: 2022
Debug Data Dirs: Type  Size     VA  Pointer
CODEVIEW    49,  e2c0,    cac0 RSDS - GUID: {29D85193-1C9D-4997-95BA-DD190FA3C1BF}
Age: 1, Pdb: C:\Work\AWMA\FakeModule\x64\Release\winspool.pdb
??    10,  e30c,    cb0c [Data not mapped]
Symbol Type: DEFERRED - No error - symbol load deferred
Load Report: no symbols loaded

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to stack trace collection we often are interested in Module Collection (we called this pattern initially Vendor Collection), especially if we would like to check if some vendor DLL is present in some process address space in a complete memory dump (kernel module list or module list from a process memory dump is trivial). Or we need to check for some vendor information from problem description (lmv command). If we have a complete memory dump from x64 system then listing modules for each process is not enough. For example, we might have this:

0: kd> lmu
start             end                 module name
00000000`00ab0000 00000000`00ae8000   AppA    (deferred)
00000000`74fe0000 00000000`7502e000   wow64win   (deferred)
00000000`75030000 00000000`75075000   wow64      (deferred)
00000000`750c0000 00000000`750c9000   wow64cpu   (deferred)
00000000`77b70000 00000000`77cf7000   ntdll      (pdb symbols)

AppA is a 32-bit process and has an additional 32-bit module list that is more useful. We can set x86 context for a thread from that process and get the list of 32-bit modules:

0: kd> .load wow64exts

0: kd> .thread /w fffffa800e372060
Implicit thread is now fffffa80`0e372060
x86 context set

0: kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols

0: kd:x86> lmu
start             end                 module name
00000000`00ab0000 00000000`00ae8000   AppA    (deferred)
00000000`73490000 00000000`73515000   COMCTL32   (deferred)
00000000`73520000 00000000`735c3000   MSVCR90    (deferred)
00000000`735d0000 00000000`7365e000   MSVCP90    (deferred)
00000000`74920000 00000000`7493e000   USERENV    (deferred)
00000000`74940000 00000000`74ade000   comctl32_74940000   (deferred)
00000000`74af0000 00000000`74b02000   MSASN1     (deferred)
00000000`74b10000 00000000`74c03000   CRYPT32    (deferred)
00000000`74dc0000 00000000`74e5b000   MSVCR80    (deferred)
00000000`74f60000 00000000`74fd6000   NETAPI32   (deferred)
00000000`74fe0000 00000000`7502e000   wow64win   (deferred)
00000000`75030000 00000000`75075000   wow64      (deferred)
00000000`750b0000 00000000`750ba000   WTSAPI32   (deferred)
00000000`750c0000 00000000`750c9000   wow64cpu   (deferred)
00000000`75cf0000 00000000`75d50000   Secur32    (deferred)
00000000`75d50000 00000000`76861000   SHELL32    (deferred)
00000000`76a10000 00000000`76aa0000   GDI32      (deferred)
00000000`76b30000 00000000`76b90000   IMM32      (deferred)
00000000`76be0000 00000000`76cf0000   kernel32   (deferred)
00000000`76e30000 00000000`76f75000   ole32      (deferred)
00000000`76f80000 00000000`7702a000   msvcrt     (deferred)
00000000`77030000 00000000`77037000   PSAPI      (deferred)
00000000`77040000 00000000`77110000   USER32     (deferred)
00000000`77110000 00000000`77169000   SHLWAPI    (deferred)
00000000`77170000 00000000`771ed000   USP10      (deferred)
00000000`77380000 00000000`7740d000   OLEAUT32   (deferred)
00000000`77640000 00000000`77649000   LPK        (deferred)
00000000`776e0000 00000000`777d0000   RPCRT4     (deferred)
00000000`777d0000 00000000`77898000   MSCTF      (deferred)
00000000`778a0000 00000000`77966000   ADVAPI32   (deferred)
00000000`77b70000 00000000`77cf7000   ntdll      (pdb symbols)
00000000`77d30000 00000000`77e90000   ntdll_77d30000 # (pdb symbols)

So it looks like we need to dump modules for each thread. However, the output would be enormous unless we skip threads having the same PID. After some tinkering I wrote this WinDbg script with moderate output volume:

.load wow64exts
!for_each_thread ".thread @#Thread; .if (@$t0 != @@c++(@$thread->Cid.UniqueProcess)) {.reload /user;lmvu;.thread /w @#Thread;.reload /user;lmvu;r $t0 = @@c++(@$thread->Cid.UniqueProcess);.effmach AMD64; }"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Looks like Windows 8 reuses the debugging concept of a frozen thread for the so called a “deeply frozen” process:

0: kd> !sprocess 2
Dumping Session 2
[...]
PROCESS fffffa8002cb2940
SessionId: 2  Cid: 0c80    Peb: 7f6c41dd000  ParentCid: 0288
DeepFreeze
DirBase: 2ef45000  ObjectTable: fffff8a002f215c0  HandleCount: <Data Not Accessible>
Image: iexplore.exe
[…]

0: kd> dt nt!_KPROCESS fffffa8002cb2940
+0x000 Header           : _DISPATCHER_HEADER
+0x018 ProfileListHead  : _LIST_ENTRY [ 0xfffffa80`02cb2958 - 0xfffffa80`02cb2958 ]
+0x028 DirectoryTableBase : 0x2ef45000
+0x030 ThreadListHead   : _LIST_ENTRY [ 0xfffffa80`01e4edf8 - 0xfffffa80`01f5bbf8 ]
+0x040 ProcessLock      : 0
+0x044 Spare0           : 0
+0x048 Affinity         : _KAFFINITY_EX
+0x0f0 ReadyListHead    : _LIST_ENTRY [ 0xfffffa80`02cb2a30 - 0xfffffa80`02cb2a30 ]
+0x100 SwapListEntry    : _SINGLE_LIST_ENTRY
+0x108 ActiveProcessors : _KAFFINITY_EX
+0x1b0 AutoAlignment    : 0y0
+0x1b0 DisableBoost     : 0y0
+0x1b0 DisableQuantum   : 0y0
+0x1b0 AffinitySet      : 0y0
+0×1b0 DeepFreeze       : 0y1
+0×1b0 TimerVirtualization : 0y1
+0×1b0 ActiveGroupsMask : 0y00000000000000000001 (0×1)
+0×1b0 ReservedFlags    : 0y000000 (0)
+0×1b0 ProcessFlags     : 0n112
+0×1b4 BasePriority     : 8 ”
+0×1b5 QuantumReset     : 6 ”
+0×1b6 Visited          : 0 ”
+0×1b7 Flags            : _KEXECUTE_OPTIONS
+0×1b8 ThreadSeed       : [20] 0
+0×208 IdealNode        : [20] 0
+0×230 IdealGlobalNode  : 0
+0×232 Spare1           : 0
+0×234 StackCount       : _KSTACK_COUNT
+0×238 ProcessListEntry : _LIST_ENTRY [ 0xfffffa80`03816b78 - 0xfffffa80`02cc2b78 ]
+0×248 CycleTime        : 0×225078
+0×250 ContextSwitches  : 0×22
+0×258 SchedulingGroup  : (null)
+0×260 FreezeCount      : 0
+0×264 KernelTime       : 0
+0×268 UserTime         : 0
+0×26c LdtFreeSelectorHint : 0
+0×26e LdtTableLength   : 0
+0×270 LdtSystemDescriptor : _KGDTENTRY64
+0×280 LdtBaseAddress   : (null)
+0×288 LdtProcessLock   : _FAST_MUTEX
+0×2c0 InstrumentationCallback : (null)

We also see that all its threads have a freeze count 1:

0: kd> !process fffffa8002cb2940 2
[...]
THREAD fffffa8001e4eb00  Cid 0c80.0514  Teb: 000007f6c41de000 Win32Thread: fffff901000e5b90 WAIT: (Suspended) KernelMode Non-Alertable
FreezeCount 1
fffffa8001e4ede0  NotificationEvent

THREAD fffffa800219c080  Cid 0c80.0d88  Teb: 000007f6c41db000 Win32Thread: fffff90103f206e0 WAIT: (Suspended) KernelMode Non-Alertable
FreezeCount 1
fffffa800219c360  NotificationEvent
[…]

0: kd> dt _KTHREAD fffffa800219c080
nt!_KTHREAD
+0x000 Header           : _DISPATCHER_HEADER
+0x018 SListFaultAddress : (null)
+0x020 QuantumTarget    : 0x18c26200
+0x028 InitialStack     : 0xfffff880`1548ddd0 Void
+0x030 StackLimit       : 0xfffff880`15488000 Void
+0x038 StackBase        : 0xfffff880`1548e000 Void
+0x040 ThreadLock       : 0
+0x048 CycleTime        : 0x15ca97c8
+0x050 CurrentRunTime   : 0
+0x054 ExpectedRunTime  : 0xd77e
+0x058 KernelStack      : 0xfffff880`1548d430 Void
+0x060 StateSaveArea    : 0xfffff880`1548de00 _XSAVE_FORMAT
+0x068 SchedulingGroup  : (null)
+0x070 WaitRegister     : _KWAIT_STATUS_REGISTER
+0x071 Running          : 0 ''
+0x072 Alerted          : [2]  ""
+0x074 KernelStackResident : 0y1
+0x074 ReadyTransition  : 0y0
+0x074 ProcessReadyQueue : 0y0
+0x074 WaitNext         : 0y0
+0x074 SystemAffinityActive : 0y0
+0x074 Alertable        : 0y0
+0x074 CodePatchInProgress : 0y0
+0x074 UserStackWalkActive : 0y0
+0x074 ApcInterruptRequest : 0y0
+0x074 QuantumEndMigrate : 0y0
+0x074 UmsDirectedSwitchEnable : 0y0
+0x074 TimerActive      : 0y0
+0x074 SystemThread     : 0y0
+0x074 ProcessDetachActive : 0y0
+0x074 CalloutActive    : 0y0
+0x074 ScbReadyQueue    : 0y0
+0x074 ApcQueueable     : 0y1
+0x074 ReservedStackInUse : 0y0
+0x074 UmsPerformingSyscall : 0y0
+0x074 Reserved         : 0y0000000000000 (0)
+0x074 MiscFlags        : 0n65537
+0x078 AutoAlignment    : 0y0
+0x078 DisableBoost     : 0y0
+0x078 UserAffinitySet  : 0y0
+0x078 AlertedByThreadId : 0y0
+0x078 QuantumDonation  : 0y0
+0x078 EnableStackSwap  : 0y1
+0x078 GuiThread        : 0y1
+0x078 DisableQuantum   : 0y0
+0x078 ChargeOnlyGroup  : 0y0
+0x078 DeferPreemption  : 0y0
+0x078 QueueDeferPreemption : 0y0
+0x078 ForceDeferSchedule : 0y0
+0x078 ExplicitIdealProcessor : 0y0
+0×078 FreezeCount      : 0y1
+0×078 EtwStackTraceApcInserted : 0y00000000 (0)
+0×078 ReservedFlags    : 0y0000000000 (0)
+0×078 ThreadFlags      : 0n8288
+0×07c Spare0           : 0
+0×080 SystemCallNumber : 0×87
+0×084 Spare1           : 0
+0×088 FirstArgument    : 0×00000000`0000017c Void
+0×090 TrapFrame        : (null)
+0×098 ApcState         : _KAPC_STATE
+0×098 ApcStateFill     : [43]  “???”
+0×0c3 Priority         : 8 ”
+0×0c4 UserIdealProcessor : 1
+0×0c8 WaitStatus       : 0n256
+0×0d0 WaitBlockList    : 0xfffffa80`0219c1c0 _KWAIT_BLOCK
+0×0d8 WaitListEntry    : _LIST_ENTRY [ 0xfffffa80`0418a458 - 0xfffff880`009eb300 ]
+0×0d8 SwapListEntry    : _SINGLE_LIST_ENTRY
+0×0e8 Queue            : 0xfffffa80`03da4bc0 _KQUEUE
+0×0f0 Teb              : 0×000007f6`c41db000 Void
+0×0f8 RelativeTimerBias : 0×00000001`8b165f54
+0×100 Timer            : _KTIMER
+0×140 WaitBlock        : [4] _KWAIT_BLOCK
+0×140 WaitBlockFill4   : [20]  “h???”
+0×154 ContextSwitches  : 0×1817
+0×140 WaitBlockFill5   : [68]  “h???”
+0×184 State            : 0×5 ”
+0×185 NpxState         : 1 ”
+0×186 WaitIrql         : 0 ”
+0×187 WaitMode         : 0 ”
+0×140 WaitBlockFill6   : [116]  “h???”
+0×1b4 WaitTime         : 0xf0172e
+0×140 WaitBlockFill7   : [164]  “h???”
+0×1e4 KernelApcDisable : 0n0
+0×1e6 SpecialApcDisable : 0n0
+0×1e4 CombinedApcDisable : 0
+0×140 WaitBlockFill8   : [40]  “h???”
+0×168 ThreadCounters   : (null)
+0×140 WaitBlockFill9   : [88]  “h???”
+0×198 XStateSave       : (null)
+0×140 WaitBlockFill10  : [136]  “h???”
+0×1c8 Win32Thread      : 0xfffff901`03f206e0 Void
+0×140 WaitBlockFill11  : [176]  “h???”
+0×1f0 Ucb              : (null)
+0×1f8 Uch              : (null)
+0×200 TebMappedLowVa   : (null)
+0×208 QueueListEntry   : _LIST_ENTRY [ 0xfffffa80`02ccf408 - 0xfffffa80`03da4bf0 ]
+0×218 NextProcessor    : 0
+0×21c DeferredProcessor : 1
+0×220 Process          : 0xfffffa80`02cb2940 _KPROCESS
+0×228 UserAffinity     : _GROUP_AFFINITY
+0×228 UserAffinityFill : [10]  “???”
+0×232 PreviousMode     : 1 ”
+0×233 BasePriority     : 8 ”
+0×234 PriorityDecrement : 0 ”
+0×234 ForegroundBoost  : 0y0000
+0×234 UnusualBoost     : 0y0000
+0×235 Preempted        : 0 ”
+0×236 AdjustReason     : 0 ”
+0×237 AdjustIncrement  : 0 ”
+0×238 Affinity         : _GROUP_AFFINITY
+0×238 AffinityFill     : [10]  “???”
+0×242 ApcStateIndex    : 0 ”
+0×243 WaitBlockCount   : 0×1 ”
+0×244 IdealProcessor   : 1
+0×248 ApcStatePointer  : [2] 0xfffffa80`0219c118 _KAPC_STATE
+0×258 SavedApcState    : _KAPC_STATE
+0×258 SavedApcStateFill : [43]  “???”
+0×283 WaitReason       : 0×5 ”
+0×284 SuspendCount     : 0 ”
+0×285 Saturation       : 0 ”
+0×286 SListFaultCount  : 0
+0×288 SchedulerApc     : _KAPC
+0×288 SchedulerApcFill0 : [1]  “??????”
+0×289 ResourceIndex    : 0×1 ”
+0×288 SchedulerApcFill1 : [3]  “???”
+0×28b QuantumReset     : 0×6 ”
+0×288 SchedulerApcFill2 : [4]  “???”
+0×28c KernelTime       : 7
+0×288 SchedulerApcFill3 : [64]  “???”
+0×2c8 WaitPrcb         : (null)
+0×288 SchedulerApcFill4 : [72]  “???”
+0×2d0 LegoData         : (null)
+0×288 SchedulerApcFill5 : [83]  “???”
+0×2db CallbackNestingLevel : 0 ”
+0×2dc UserTime         : 0xa
+0×2e0 SuspendEvent     : _KEVENT
+0×2f8 ThreadListEntry  : _LIST_ENTRY [ 0xfffffa80`01c41378 - 0xfffffa80`01e4edf8 ]
+0×308 MutantListHead   : _LIST_ENTRY [ 0xfffffa80`0219c388 - 0xfffffa80`0219c388 ]
+0×318 ReadOperationCount : 0n392
+0×320 WriteOperationCount : 0n321
+0×328 OtherOperationCount : 0n240
+0×330 ReadTransferCount : 0n1849338
+0×338 WriteTransferCount : 0n1197496
+0×340 OtherTransferCount : 0n4972

This is different when a process is under a debugger and all its threads are frozen except the one that communicates to the debugger like in this case study. In Windows 8 this happens, for example, when we switch to a desktop from IE launched from the start page. Then we would see shortly that iexplore.exe process changes from Running to Suspended in Task Manager Details page. We call this pattern Frozen Process to cover both the new feature and a debugged process case.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Very useful pattern for the analysis of memory dumps from terminal services environments is Incomplete Session. Normally the session processes include csrss.exe, winlogon.exe, wfshell.exe (in case of Citrix), explorer.exe and a few user defined processes such as winword.exe, for example:

0: kd> !session
Sessions on machine: 6
Valid Sessions: 0 1 3 5 6 8

0: kd> !sprocess 6
Dumping Session 6

_MM_SESSION_SPACE fffffa6009447000
_MMSESSION        fffffa6009447b40
PROCESS fffffa800fcee630
SessionId: 6  Cid: 1974    Peb: 7fffffd5000  ParentCid: 147c
DirBase: 158baf000  ObjectTable: fffff8801ef13b00  HandleCount: 532.
Image: csrss.exe

PROCESS fffffa800fc77040
SessionId: 6  Cid: 1ae4    Peb: 7fffffde000  ParentCid: 147c
DirBase: 15d2b4000  ObjectTable: fffff8802084b570  HandleCount: 238.
Image: winlogon.exe

PROCESS fffffa800fe61040
SessionId: 6  Cid: 1edc    Peb: 7efdf000  ParentCid: 1ec8
DirBase: 14df74000  ObjectTable: fffff88020f486e0  HandleCount: 313.
Image: wfshell.exe

PROCESS fffffa800ff5a660
SessionId: 6  Cid: 2054    Peb: 7fffffdf000  ParentCid: 1dbc
DirBase: 201a81000  ObjectTable: fffff88020dd56e0  HandleCount: 447.
Image: explorer.exe

PROCESS fffffa800fe28040
SessionId: 6  Cid: 1ce4    Peb: 7efdf000  ParentCid: 13a8
DirBase: 11f552000  ObjectTable: fffff8801fe96990  HandleCount: 1842.
Image: WINWORD.EXE

PROCESS fffffa800f119c10
SessionId: 6  Cid: 2074    Peb: 7efdf000  ParentCid: 2054
DirBase: 2d994f000  ObjectTable: fffff8801e76aec0  HandleCount: 673.
Image: iexplore.exe

If we compare with the last session #8 we see that the latter has only 2 processes:

0: kd> !sprocess 8
Dumping Session 8

_MM_SESSION_SPACE fffffa600bafc000
_MMSESSION        fffffa600bafcb40
PROCESS fffffa80103a4480
SessionId: 8  Cid: 2858    Peb: 7fffffdf000  ParentCid: 2660
DirBase: a04bb000  ObjectTable: fffff8801cb926a0  HandleCount: 534.
Image: csrss.exe

PROCESS fffffa801065b770
SessionId: 8  Cid: 2878    Peb: 7fffffdf000  ParentCid: 2660
DirBase: 5da40000  ObjectTable: fffff8801ce5e440  HandleCount: 235.
Image: winlogon.exe

Such anomalies may point to a disconnected session that failed to terminate due to some unresponsive session process or a session that is stuck in session initialization process launch sequence due to threads blocked in wait chains so process threads need to be analyzed.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we dump a stack trace collection from a complete or kernel memory dump we see lots of synchronization and notification events, for example:

THREAD fffffa8003d33120  Cid 0734.0868  Teb: 000007fffffd4000 Win32Thread: fffff900c07182e0 WAIT: (UserRequest) UserMode Alertable
fffffa8003413d20  NotificationEvent
fffffa80020b5170  NotificationEvent
fffffa80017f31e0  NotificationEvent
fffffa80013f8cf0  NotificationEvent
fffffa8002547ee0  NotificationEvent
fffffa8002547e80  NotificationEvent
fffffa8004186100  NotificationEvent
fffffa8003dcfa80  NotificationEvent
fffffa8003df6870  NotificationEvent
fffffa8003bbd5e0  NotificationEvent
fffffa8003b5d4e0  NotificationEvent
fffffa800390c690  NotificationEvent
fffffa8003dbc410  NotificationEvent
fffffa8003dbc3b0  NotificationEvent
fffffa8004041040  NotificationEvent
fffffa8003dde8a0  NotificationEvent
fffffa80038f4530  NotificationEvent
fffffa800401fa50  NotificationEvent
fffffa800398a550  NotificationEvent
fffffa8003b587e0  NotificationEvent
fffffa800398d200  SynchronizationEvent
IRP List:
fffffa800150b010: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa80015c1010: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa80017f53a0: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa80014ccca0: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa80011fa710: (0006,03a0) Flags: 00060000  Mdl: 00000000
fffffa80011d6070: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa80030b5450: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa8004149810: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa800419b500: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa80040c2520: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa8003b75520: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa8004082ca0: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa8004082010: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa800403aa40: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa800403a010: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa8003dbc9d0: (0006,0358) Flags: 00060000  Mdl: 00000000
fffffa8003dbc010: (0006,0358) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 fffff88001697690
Owning Process            fffffa80039bac10       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      41897          Ticks: 561 (0:00:00:08.765)
Context Switch Count      4744                 LargeStack
UserTime                  00:00:00.187
KernelTime                00:00:01.218
Win32 Start Address SHLWAPI!WrapperThreadProc (0×000007fefe854f20)
Stack Init fffff9800ef2fdb0 Current fffff9800ef2f260
Base fffff9800ef30000 Limit fffff9800ef27000 Call 0
Priority 12 BasePriority 8 PriorityDecrement 2 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff980`0ef2f2a0 fffff800`0185d695 nt!KiSwapContext+0×84
fffff980`0ef2f3e0 fffff800`0185ad2f nt!KiSwapThread+0×125
fffff980`0ef2f440 fffff800`01ac1813 nt!KeWaitForMultipleObjects+0×703
fffff980`0ef2f4b0 fffff800`01ac1a03 nt!ObpWaitForMultipleObjects+0×216
fffff980`0ef2f960 fffff800`0184dcf3 nt!NtWaitForMultipleObjects+0xe2
fffff980`0ef2fbb0 00000000`776f082a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff980`0ef2fc20)
00000000`03acf718 00000000`774ced73 ntdll!NtWaitForMultipleObjects+0xa
00000000`03acf720 00000000`775ee97d kernel32!WaitForMultipleObjectsEx+0×10b
00000000`03acf830 00000000`775ee86e USER32!RealMsgWaitForMultipleObjectsEx+0×129
00000000`03acf8d0 000007fe`fed47a9a USER32!MsgWaitForMultipleObjectsEx+0×46
00000000`03acf910 000007fe`fe854d48 SHELL32!CChangeNotify::ThreadProc+0xba
00000000`03acfb90 00000000`774ccdcd SHLWAPI!WrapperThreadProc+0xfc
00000000`03acfc70 00000000`776ec6e1 kernel32!BaseThreadInitThunk+0xd
00000000`03acfca0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The problem with such synchronization objects as events is that they don’t have an owner field as in other structures:

1: kd> dt -r _KEVENT
ntdll!_KEVENT
+0x000 Header           : _DISPATCHER_HEADER
+0x000 Type             : UChar
+0x001 Abandoned        : UChar
+0x001 Absolute         : UChar
+0x001 NpxIrql          : UChar
+0x001 Signalling       : UChar
+0x002 Size             : UChar
+0x002 Hand             : UChar
+0x003 Inserted         : UChar
+0x003 DebugActive      : UChar
+0x003 DpcActive        : UChar
+0x000 Lock             : Int4B
+0x004 SignalState      : Int4B
+0x008 WaitListHead     : _LIST_ENTRY
+0x000 Flink            : Ptr64 _LIST_ENTRY
+0x008 Blink            : Ptr64 _LIST_ENTRY

1: kd> dt _KMUTANT
nt!_KMUTANT
+0x000 Header           : _DISPATCHER_HEADER
+0x018 MutantListEntry  : _LIST_ENTRY
+0×028 OwnerThread      : Ptr64 _KTHREAD
+0×030 Abandoned        : UChar
+0×031 ApcDisable       : UChar

Fortunately many of such events are created to wait for asynchronous I/O and their addresses are stored in IRP structures that also have an associated thread. For example, in the thread above we find one of notification events, fffffa80020b5170, in an IRP fffffa800150b010:

1: kd> !irp fffffa800150b010 -v
Irp is active with 9 stacks 9 is current (= 0xfffffa800150b320)
No Mdl: No System Buffer: Thread fffffa8003d33120:  Irp stack trace.
Flags = 00060000
ThreadListEntry.Flink = fffffa80015c1030
ThreadListEntry.Blink = fffffa8003d334d8
IoStatus.Status = 00000000
IoStatus.Information = fffff88002ac09c0
RequestorMode = 00000001
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 04127ca0
UserEvent = fffffa80020b5170
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 04127ca0
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = fffff80001912fc0   nt!FsRtlCancelNotify
UserBuffer = 04127498
&Tail.Overlay.DeviceQueueEntry = fffffa800150b088
Tail.Overlay.Thread = fffffa8003d33120
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = fffff88002ac0a00
Tail.Overlay.ListEntry.Blink = fffff88002ac0a00
Tail.Overlay.CurrentStackLocation = fffffa800150b320
Tail.Overlay.OriginalFileObject = fffffa8002390770
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
>[  c, 2]   0  1 fffffa8001bcb030 fffffa8002390770 00000000-00000000    pending
\FileSystem\Ntfs
Args: 00000800 00000015 00000000 00000000

Another example when IRP fffffa80022d4410 was created by one thread fffffa8002119700 but another thread fffffa8001fda450 is waiting for its associated event fffffa8002093190:

1: kd> !irpfind
Irp    [ Thread ] irpStack: (Mj,Mn)   DevObj  [Driver]         MDL Process
fffffa8002f2d980 [00000000] irpStack: ( f, 0)  fffffa8002f6f050 [ \Driver\usbehci]
fffffa800200dc40 [fffffa80030a0710] irpStack: ( d, 0)  fffffa8001f192d0 [ \FileSystem\Npfs]
fffffa800203d280 [fffffa80035b1b10] irpStack: ( e, 0)  fffffa8001c09b50 [ \Driver\netbt] 0xfffffa8002f10970
fffffa800228e4e0 [00000000] irpStack: ( e, 0)  fffffa8002be1840 [ \Driver\CmBatt]
fffffa800229c6a0 [00000000] irpStack: ( f, 0)  fffffa8002f2c050 [ \Driver\usbuhci]
fffffa800229a6a0 [00000000] irpStack: ( f, 0)  fffffa8002f2c050 [ \Driver\usbuhci]
fffffa80022946a0 [00000000] irpStack: ( f, 0)  fffffa8002f2c050 [ \Driver\usbuhci]
fffffa80022a28a0 [00000000] irpStack: ( f, 0)  fffffa80022d04c0 [ \Driver\usbhub]
fffffa80022a6230 [00000000] irpStack: ( f, 0)  fffffa8002f2c050 [ \Driver\usbuhci]
fffffa80022d4410 [fffffa8002119700] irpStack: ( e,2d)  fffffa80022ee720 [ \Driver\AFD]
[…]

1: kd> !irp fffffa80022d4410 -v
Irp is active with 4 stacks 4 is current (= 0xfffffa80022d45b8)
No Mdl: System buffer=fffffa8001fda150: Thread fffffa8002119700:  Irp stack trace.
Flags = 00060030
ThreadListEntry.Flink = fffffa8002396c80
ThreadListEntry.Blink = fffffa8002119ab8
IoStatus.Status = 00000000
IoStatus.Information = 00000000
RequestorMode = 00000001
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 7fefc914858
UserEvent = fffffa8002093190
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 7fefc914858
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = fffff980044ad250   afd!AfdCancelAddressListChange
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = fffffa80022d4488
Tail.Overlay.Thread = fffffa8002119700
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = 00000000
Tail.Overlay.ListEntry.Blink = 00000000
Tail.Overlay.CurrentStackLocation = fffffa80022d45b8
Tail.Overlay.OriginalFileObject = fffffa8002bbe050
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
>[  e,2d]   5  1 fffffa80022ee720 fffffa8002bbe050 00000000-00000000    pending
\Driver\AFD
Args: fffffa8002bbef50 fffffa8002bbef50 fffff9800446cf00 fffffa800250d220

THREAD fffffa8002119700  Cid 0310.0318  Teb: 000007fffffdc000 Win32Thread: fffff900c07b6d60 WAIT: (DelayExecution) UserMode Non-Alertable
fffffa80021197b8  NotificationTimer
IRP List:
fffffa80022d4410: (0006,03a0) Flags: 00060030  Mdl: 00000000
fffffa8002396c60: (0006,03a0) Flags: 00060030  Mdl: 00000000
Not impersonating
DeviceMap                 fffff880017b4c70
Owning Process            fffffa800209f880       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      42336          Ticks: 122 (0:00:00:01.906)
Context Switch Count      159                 LargeStack
UserTime                  00:00:00.031
KernelTime                00:00:00.140
Win32 Start Address ADVAPI32!ScSvcctrlThreadW (0×000007fefe2b4bd0)
Stack Init fffff9800ccabdb0 Current fffff9800ccab990
Base fffff9800ccac000 Limit fffff9800cca6000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff980`0ccab9d0 fffff800`0185d695 nt!KiSwapContext+0×84
fffff980`0ccabb10 fffff800`0185bbe9 nt!KiSwapThread+0×125
fffff980`0ccabb70 fffff800`01a8b1cd nt!KeDelayExecutionThread+0×339
fffff980`0ccabbf0 fffff800`0184dcf3 nt!NtDelayExecution+0×5c
fffff980`0ccabc20 00000000`776f05ba nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff980`0ccabc20)
00000000`000ff9d8 00000000`774cd908 ntdll!NtDelayExecution+0xa
00000000`000ff9e0 000007fe`fc8ba8c0 kernel32!SleepEx+0×84
00000000`000ffa60 000007fe`fc8b17bd rpcss!ObjectExporterWorkerThread+0×50b
00000000`000ffb30 000007fe`fc8b27f2 rpcss!ScmServiceMain+0xe4
00000000`000ffb60 00000000`ffaa1771 rpcss!ServiceMain+0×251
00000000`000ffe20 000007fe`fe2b4bf5 svchost!ServiceStarter+0×1ea
00000000`000ffeb0 00000000`774ccdcd ADVAPI32!ScSvcctrlThreadW+0×25
00000000`000ffee0 00000000`776ec6e1 kernel32!BaseThreadInitThunk+0xd
00000000`000fff10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa8001fda450  Cid 0310.031c  Teb: 000007fffffda000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Alertable
fffffa8002012ef0  SynchronizationTimer
fffffa800208d7a0  SynchronizationEvent
fffffa80023bb820  SynchronizationEvent
fffffa80023bb740  SynchronizationEvent
fffffa8001fd9730  SynchronizationEvent
fffffa8002093190  SynchronizationEvent
fffffa8001a0eee0  SynchronizationEvent
Not impersonating
DeviceMap                 fffff880017b4c70
Owning Process            fffffa800209f880       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      5767           Ticks: 36691 (0:00:09:33.296)
Context Switch Count      8
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWaiterpThread (0×00000000776c6930)
Stack Init fffff9800d2c0db0 Current fffff9800d2c0260
Base fffff9800d2c1000 Limit fffff9800d2bb000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff980`0d2c02a0 fffff800`0185d695 nt!KiSwapContext+0×84
fffff980`0d2c03e0 fffff800`0185ad2f nt!KiSwapThread+0×125
fffff980`0d2c0440 fffff800`01ac1813 nt!KeWaitForMultipleObjects+0×703
fffff980`0d2c04b0 fffff800`01ac1a03 nt!ObpWaitForMultipleObjects+0×216
fffff980`0d2c0960 fffff800`0184dcf3 nt!NtWaitForMultipleObjects+0xe2
fffff980`0d2c0bb0 00000000`776f082a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff980`0d2c0c20)
00000000`00d1fb08 00000000`776c6b07 ntdll!NtWaitForMultipleObjects+0xa
00000000`00d1fb10 00000000`774ccdcd ntdll!TppWaiterpThread+0×14d
00000000`00d1fdb0 00000000`776ec6e1 kernel32!BaseThreadInitThunk+0xd
00000000`00d1fde0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a high contention pattern variant where the contention is around a monitor object. For example, we have a distributed CPU spike for some threads:

0:000> !runaway
 User Mode Time
  Thread       Time
   9:6ff4      0 days 0:07:39.019
  12:6b88      0 days 0:06:19.786
  11:6bf0      0 days 0:06:13.889
  10:6930      0 days 0:06:09.240
  16:3964      0 days 0:05:44.483
  17:6854      0 days 0:05:35.326
  13:668c      0 days 0:05:35.123
  14:5594      0 days 0:05:34.858
  15:7248      0 days 0:05:23.111
   2:c54       0 days 0:00:41.215
   4:1080      0 days 0:00:00.349
   7:10f0      0 days 0:00:00.302
   0:c3c       0 days 0:00:00.271
[...]

If we look at their stack traces we find them all blocked trying to enter a monitor, for example:

0:000> ~*k

[...]

  12  Id: d50.6b88 Suspend: 0 Teb: 000007ff`fffd8000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`1a98e798 000007fe`fd0c1420 ntdll!ZwWaitForMultipleObjects+0xa
00000000`1a98e7a0 00000000`76e82cf3 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`1a98e8a0 000007fe`f82e0669 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`1a98e930 000007fe`f82dbec9 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
00000000`1a98e9d0 000007fe`f82a0569 mscorwks!Thread::DoAppropriateAptStateWait+0x41
00000000`1a98ea30 000007fe`f82beaec mscorwks!Thread::DoAppropriateWaitWorker+0x191
00000000`1a98eb30 000007fe`f81f1b9a mscorwks!Thread::DoAppropriateWait+0x5c
00000000`1a98eba0 000007fe`f82fd3c9 mscorwks!CLREvent::WaitEx+0xbe
00000000`1a98ec50 000007fe`f81ac6be mscorwks!AwareLock::EnterEpilog+0xc9
00000000`1a98ed20 000007fe`f81c7b2b mscorwks!AwareLock::Enter+0x72
00000000`1a98ed50 000007fe`f87946af mscorwks!AwareLock::Contention+0x1fb
00000000`1a98ee20 000007ff`00161528 mscorwks!JITutil_MonContention+0xdf
00000000`1a98efd0 000007ff`0016140e 0×7ff`00161528
00000000`1a98f040 000007ff`00167271 0×7ff`0016140e
00000000`1a98f0a0 000007fe`f74e2bbb 0×7ff`00167271
00000000`1a98f130 000007fe`f753ed76 mscorlib_ni+0×2f2bbb
00000000`1a98f180 000007fe`f8390282 mscorlib_ni+0×34ed76
00000000`1a98f1d0 000007fe`f8274363 mscorwks!CallDescrWorker+0×82
00000000`1a98f220 000007fe`f8274216 mscorwks!CallDescrWorkerWithHandler+0xd3
00000000`1a98f2c0 000007fe`f81c96a7 mscorwks!DispatchCallDebuggerWrapper+0×3e
00000000`1a98f320 000007fe`f830ae42 mscorwks!DispatchCallNoEH+0×5f
00000000`1a98f3a0 000007fe`f81bdc00 mscorwks!AddTimerCallback_Worker+0×92
00000000`1a98f430 000007fe`f82a41a5 mscorwks!ManagedThreadCallState::IsAppDomainEqual+0×4c
00000000`1a98f480 000007fe`f82df199 mscorwks!SVR::gc_heap::make_heap_segment+0×155
00000000`1a98f550 000007fe`f82ececa mscorwks!DoOpenIAssemblyStress::DoOpenIAssemblyStress+0×99
00000000`1a98f590 000007fe`f830c0db mscorwks!AddTimerCallbackEx+0xba
00000000`1a98f650 000007fe`f81ebb37 mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×53
00000000`1a98f6b0 000007fe`f81fe92a mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×157
00000000`1a98f750 000007fe`f81bb1fc mscorwks!ThreadpoolMgr::WorkerThreadStart+0×1ba
00000000`1a98f7f0 00000000`76e7652d mscorwks!Thread::intermediateThreadProc+0×78
00000000`1a98fcc0 00000000`76fac521 kernel32!BaseThreadInitThunk+0xd
00000000`1a98fcf0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

[...]

  15  Id: d50.7248 Suspend: 0 Teb: 000007ff`ffee6000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`1c16e6f0 000007fe`f87946af mscorwks!AwareLock::Contention+0×13b
00000000`1c16e7c0 000007ff`0016135e mscorwks!JITutil_MonContention+0xdf
00000000`1c16e970 000007ff`0016726b 0×7ff`0016135e
00000000`1c16e9c0 000007fe`f74e2bbb 0×7ff`0016726b
00000000`1c16ea50 000007fe`f753ed76 mscorlib_ni+0×2f2bbb
00000000`1c16eaa0 000007fe`f8390282 mscorlib_ni+0×34ed76
00000000`1c16eaf0 000007fe`f8274363 mscorwks!CallDescrWorker+0×82
00000000`1c16eb40 000007fe`f8274216 mscorwks!CallDescrWorkerWithHandler+0xd3
00000000`1c16ebe0 000007fe`f81c96a7 mscorwks!DispatchCallDebuggerWrapper+0×3e
00000000`1c16ec40 000007fe`f830ae42 mscorwks!DispatchCallNoEH+0×5f
00000000`1c16ecc0 000007fe`f81bdc00 mscorwks!AddTimerCallback_Worker+0×92
00000000`1c16ed50 000007fe`f82a41a5 mscorwks!ManagedThreadCallState::IsAppDomainEqual+0×4c
00000000`1c16eda0 000007fe`f82df199 mscorwks!SVR::gc_heap::make_heap_segment+0×155
00000000`1c16ee70 000007fe`f82ececa mscorwks!DoOpenIAssemblyStress::DoOpenIAssemblyStress+0×99
00000000`1c16eeb0 000007fe`f830c0db mscorwks!AddTimerCallbackEx+0xba
00000000`1c16ef70 000007fe`f81ebb37 mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×53
00000000`1c16efd0 000007fe`f81fe92a mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×157
00000000`1c16f070 000007fe`f81bb1fc mscorwks!ThreadpoolMgr::WorkerThreadStart+0×1ba
00000000`1c16f110 00000000`76e7652d mscorwks!Thread::intermediateThreadProc+0×78
00000000`1c16f9e0 00000000`76fac521 kernel32!BaseThreadInitThunk+0xd
00000000`1c16fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

[...]

Thread #15 seems was caught at the time it was trying to enter and not waiting yet. If we check a monitor object the thread #12 tries to enter we see it has an address 01af0be8:

0:000> !u 000007ff`00161528
Normal JIT generated code
[…]
000007ff`00161505 90              nop
000007ff`00161506 48b8f089ae1100000000 mov rax,11AE89F0h
000007ff`00161510 488b00          mov     rax,qword ptr [rax]
000007ff`00161513 48894528        mov     qword ptr [rbp+28h],rax
000007ff`00161517 488b4528        mov     rax,qword ptr [rbp+28h]
000007ff`0016151b 48894518        mov     qword ptr [rbp+18h],rax
000007ff`0016151f 488b4d28        mov     rcx,qword ptr [rbp+28h]
000007ff`00161523 e8b8d422f8      call    mscorwks!JIT_MonEnter (000007fe`f838e9e0)
>>> 000007ff`00161528 90              nop
000007ff`00161529 90              nop
000007ff`0016152a 90              nop
[…]
000007ff`001615d2 4883c430        add     rsp,30h
000007ff`001615d6 5d              pop     rbp
000007ff`001615d7 f3c3            rep ret

0:000> dps 11AE89F0h l1
00000000`11ae89f0  00000000`01af0be8

This object seems to be owned by the thread #17:

0:000> !syncblk
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
 1362 000000001ba7b6c0           15         1 000000001c0173b0  6854  17   0000000001af0be8 System.Object
[…]

This thread seems to be blocked in ALPC:

0:017> k
Child-SP          RetAddr           Call Site
00000000`1d55c9e8 000007fe`fee1a776 ntdll!NtAlpcSendWaitReceivePort+0xa
00000000`1d55c9f0 000007fe`fee14e42 rpcrt4!LRPC_CCALL::SendReceive+0x156
00000000`1d55cab0 000007fe`ff0828c0 rpcrt4!I_RpcSendReceive+0x42
00000000`1d55cae0 000007fe`ff08282f ole32!ThreadSendReceive+0x40
00000000`1d55cb30 000007fe`ff08265b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3
00000000`1d55cbd0 000007fe`fef3daaa ole32!CRpcChannelBuffer::SendReceive2+0x11b
00000000`1d55cd90 000007fe`fef3da0c ole32!CAptRpcChnl::SendReceive+0x52
00000000`1d55ce60 000007fe`ff08205d ole32!CCtxComChnl::SendReceive+0x68
00000000`1d55cf10 000007fe`feebfd61 ole32!NdrExtpProxySendReceive+0x45
00000000`1d55cf40 000007fe`ff07f82f rpcrt4!NdrpClientCall2+0x9ea
00000000`1d55d6b0 000007fe`fef3d8a2 ole32!ObjectStublessClient+0x1ad
00000000`1d55da40 000007fe`fa511ba8 ole32!ObjectStubless+0x42
[...] 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was very pleased to find out this book that uses WinDbg as OS reversing tool. Not only you learn a very important aspect of Windows internals related to crash and hang memory dump analysis (all crash processing starts from memory manager) but you also learn many WinDbg commands from practical reversing experiments. I was even more pleased to find the output of WinDbg command on the page 0, before even the table of contents.

What Makes It Page?: The Windows 7 (x64) Virtual Memory Manager

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We continue with such problem pattern category and discuss Unresponsive Window pattern. The previous one was Error Message Box. We all see hang windows from time to time. This can happen, for example, from a main thread blocked in a wait chain. Some windows become unresponsive only temporary, for example, when a window message loop results in a CPU intensive window procedure code path. When I open large WinDbg logs generated by WinDbg scripts running on a complete memory dump in Notepad it opens up a frozen window for some seconds and sometimes for a minute or two. To get an unresponsive window for a longer time I opened a PDF file with a size of a few MB and I attached WinDbg. I got this stack trace:

0:000> k
Child-SP          RetAddr           Call Site
00000000`001ecce0 000007fe`ff9fdf89 USP10!otlCacheManager::GetNextLookup+0x12a
00000000`001ecd40 000007fe`ff9fa134 USP10!ApplyFeatures+0x489
00000000`001ed000 000007fe`ff9e1600 USP10!SubstituteOtlGlyphs+0x224
00000000`001ed0b0 000007fe`ff9d4b60 USP10!GenericEngineGetGlyphs+0x1000
00000000`001ed450 000007fe`ff9989c5 USP10!ShlShape+0x7a0
00000000`001ed670 000007fe`ff9a7363 USP10!ScriptShape+0x205
00000000`001ed710 000007fe`ff9a8ac9 USP10!RenderItemNoFallback+0x433
00000000`001ed7d0 000007fe`ff9a8d86 USP10!RenderItemWithFallback+0x129
00000000`001ed820 000007fe`ff9aa5f7 USP10!RenderItem+0x36
00000000`001ed870 000007fe`ff99b2c9 USP10!ScriptStringAnalyzeGlyphs+0x277
00000000`001ed910 000007fe`ff30285c USP10!ScriptStringAnalyse+0x399
00000000`001ed990 000007fe`ff3031c1 LPK!EditStringAnalyse+0x1d4
00000000`001eda70 000007fe`fc876c05 LPK!EditCchInWidth+0x4e
00000000`001edad0 000007fe`fc85862e COMCTL32!EditML_BuildchLines+0x221
00000000`001edba0 000007fe`fc878f56 COMCTL32!Edit_ResetTextInfo+0x82
00000000`001edbe0 000007fe`fc85a566 COMCTL32!EditML_WndProc+0x456
00000000`001edcd0 00000000`77a19bd1 COMCTL32!Edit_WndProc+0xe0a
00000000`001edd70 00000000`77a16aa8 USER32!UserCallWinProcCheckWow+0x1ad
00000000`001ede30 00000000`77a16bad USER32!SendMessageWorker+0x682
00000000`001edec0 00000000`ff7f4256 USER32!SendMessageW+0x5c
00000000`001edf10 00000000`ff7f43d6 NOTEPAD!LoadFile+0x7cb
00000000`001ee260 00000000`ff7f1018 NOTEPAD!NPInit+0x802
00000000`001efbb0 00000000`ff7f133c NOTEPAD!WinMain+0xc7
00000000`001efc30 00000000`7764652d NOTEPAD!DisplayNonGenuineDlgWorker+0x2da
00000000`001efcf0 00000000`77b2c521 kernel32!BaseThreadInitThunk+0xd
00000000`001efd20 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Another notepad.exe instance had this similar stack trace:

0:000> k
Child-SP          RetAddr           Call Site
00000000`0015ca60 000007fe`ff9e2152 USP10!ShapingLibraryInternal::RestoreCharMap+0x12
00000000`0015cab0 000007fe`ff9d80b8 USP10!GenericEngineGetGlyphPositions+0x2a2
00000000`0015ce60 000007fe`ff9d548e USP10!ShapingGetGlyphPositions+0x8c8
00000000`0015d030 000007fe`ff998c72 USP10!ShlPlace+0x2de
00000000`0015d1e0 000007fe`ff9a742d USP10!ScriptPlace+0x1f2
00000000`0015d270 000007fe`ff9a8ac9 USP10!RenderItemNoFallback+0x4fd
00000000`0015d330 000007fe`ff9a8d86 USP10!RenderItemWithFallback+0x129
00000000`0015d380 000007fe`ff9aa5f7 USP10!RenderItem+0x36
00000000`0015d3d0 000007fe`ff99b2c9 USP10!ScriptStringAnalyzeGlyphs+0x277
00000000`0015d470 000007fe`ff30285c USP10!ScriptStringAnalyse+0x399
00000000`0015d4f0 000007fe`ff3031c1 LPK!EditStringAnalyse+0x1d4
00000000`0015d5d0 000007fe`fc876c05 LPK!EditCchInWidth+0x4e
00000000`0015d630 000007fe`fc85862e COMCTL32!EditML_BuildchLines+0x221
00000000`0015d700 000007fe`fc878f56 COMCTL32!Edit_ResetTextInfo+0x82
00000000`0015d740 000007fe`fc85a566 COMCTL32!EditML_WndProc+0x456
00000000`0015d830 00000000`77a19bd1 COMCTL32!Edit_WndProc+0xe0a
00000000`0015d8d0 00000000`77a16aa8 USER32!UserCallWinProcCheckWow+0x1ad
00000000`0015d990 00000000`77a16bad USER32!SendMessageWorker+0x682
00000000`0015da20 00000000`ff7f4256 USER32!SendMessageW+0x5c
00000000`0015da70 00000000`ff7f43d6 NOTEPAD!LoadFile+0×7cb
00000000`0015ddc0 00000000`ff7f1018 NOTEPAD!NPInit+0×802
00000000`0015f710 00000000`ff7f133c NOTEPAD!WinMain+0xc7
00000000`0015f790 00000000`7764652d NOTEPAD!DisplayNonGenuineDlgWorker+0×2da
00000000`0015f850 00000000`77b2c521 kernel32!BaseThreadInitThunk+0xd
00000000`0015f880 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

This thread is also spiking and all work was done in a Unicode script processor as the PDF file was obviously not an ASCII text file:

0:000> !runaway f
User Mode Time
Thread       Time
0:fa0       0 days 0:00:12.402
Kernel Mode Time
Thread       Time
0:fa0       0 days 0:00:10.826
Elapsed Time
Thread       Time
0:fa0       0 days 0:00:34.654

0:000> lmv m USP10
start             end                 module name
000007fe`ff990000 000007fe`ffa59000   USP10      (pdb symbols)          c:\mss\usp10.pdb\DB4EC1196F91457FBB0A462D9D0AFEC31\usp10.pdb
Loaded symbol image file: C:\Windows\system32\USP10.dll
Image path: C:\Windows\system32\USP10.dll
Image name: USP10.dll
Timestamp:        Sat Nov 20 13:15:33 2010 (4CE7C9F5)
CheckSum:         000C4B61
ImageSize:        000C9000
File version:     1.626.7601.17514
Product version:  1.626.7601.17514
File flags:       0 (Mask 3F)
File OS:          40004 NT Win32
File type:        2.0 Dll
File date:        00000000.00000000
Translations:     0409.04b0
CompanyName:      Microsoft Corporation
ProductName:      Microsoft(R) Uniscribe Unicode script processor
InternalName:     Uniscribe
OriginalFilename: Uniscribe
ProductVersion:   1.0626.7601.17514
FileVersion:      1.0626.7601.17514 (win7sp1_rtm.101119-1850)
FileDescription:  Uniscribe Unicode script processor
LegalCopyright:   © Microsoft Corporation. All rights reserved.

We see LoadFile function and find a file name from execution residue on the raw stack:

0:000> dpu 00000000`0015da70
00000000`0015da70  00000000`00000000
00000000`0015da78  00000000`00000000
00000000`0015da80  00000000`00000000
00000000`0015da88  00000000`00000000
00000000`0015da90  00000000`02b40040 "%PDF-1.4..%µµµµ..1 0 obj..<</Type/Catalog/Pages 2 0 R/L"
00000000`0015da98  00000000`00576a62
00000000`0015daa0  00000000`00000000
00000000`0015daa8  00000000`00000000
00000000`0015dab0  00000000`025c0000
00000000`0015dab8  00000000`00000000
00000000`0015dac0  00000000`00000000
00000000`0015dac8  00000000`00000100
00000000`0015dad0  00000000`00000000
00000000`0015dad8  00000000`025c0000
00000000`0015dae0  00000000`00000265
00000000`0015dae8  00000000`ff800b40 "C:\DL\History-Russian-Literature-VIII-Volume2.pdf"
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a variant of Handled Exception pattern in kernel space (similar to user and managed spaces). The crash dump was the same as in Hidden Exception in kernel space pattern:

fffff880`0a83d910  00000000`00000000
fffff880`0a83d918  fffff6fc`40054fd8
fffff880`0a83d920  fffff880`0a83dca0
fffff880`0a83d928  fffff800`016bcc1c nt!_C_specific_handler+0xcc
fffff880`0a83d930  00000000`00000000
fffff880`0a83d938  00000000`00000000
fffff880`0a83d940  00000000`00000000
fffff880`0a83d948  00000000`00000000
fffff880`0a83d950  fffff800`0189ee38 nt!BBTBuffer <PERF> (nt+0x280e38)
fffff880`0a83d958  fffff880`0a83e940
fffff880`0a83d960  fffff800`016ad767 nt!IopCompleteRequest+0x147
fffff880`0a83d968  fffff880`0a83de40
fffff880`0a83d970  fffff800`01665e40 nt!_GSHandlerCheck_SEH
fffff880`0a83d978  fffff800`017e5338 nt!_imp_NtOpenSymbolicLinkObject+0xfe30
fffff880`0a83d980  fffff880`0a83e310
fffff880`0a83d988  00000000`00000000
fffff880`0a83d990  00000000`00000000
fffff880`0a83d998  fffff800`016b42dd nt!RtlpExecuteHandlerForException+0xd
fffff880`0a83d9a0  fffff800`017d7d0c nt!_imp_NtOpenSymbolicLinkObject+0×2804
fffff880`0a83d9a8  fffff880`0a83eab0
fffff880`0a83d9b0  00000000`00000000

0: kd> ub fffff800`016b42dd
nt!RtlpExceptionHandler+0x24:
fffff800`016b42c4 cc              int     3
fffff800`016b42c5 cc              int     3
fffff800`016b42c6 cc              int     3
fffff800`016b42c7 cc              int     3
fffff800`016b42c8 0f1f840000000000 nop     dword ptr [rax+rax]
nt!RtlpExecuteHandlerForException:
fffff800`016b42d0 4883ec28        sub     rsp,28h
fffff800`016b42d4 4c894c2420      mov     qword ptr [rsp+20h],r9
fffff800`016b42d9 41ff5130        call    qword ptr [r9+30h]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is an example of Hidden Exception pattern in kernel space:

0: kd> !thread
THREAD fffffa800d4bf9c0  Cid 0e88.56e0  Teb: 000007fffffd8000 Win32Thread: 0000000000000000 RUNNING on processor 0
Not impersonating
DeviceMap                 fffff8a001e91950
Owning Process            fffffa800b33cb30       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      13154529       Ticks: 0
Context Switch Count      1426
UserTime                  00:00:00.015
KernelTime                00:00:00.124
Win32 Start Address 0x0000000077728d20
Stack Init fffff8800a83fdb0 Current fffff8800a83eb90
Base fffff8800a840000 Limit fffff8800a83a000 Call 0
Priority 10 BasePriority 10 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
[…]

0: kd> dps fffff8800a83a000 fffff8800a840000
[...]
fffff880`0a83e180  fffff880`0a83ea10
fffff880`0a83e188  fffff880`0a83e6d0
fffff880`0a83e190  fffff880`0a83e968
fffff880`0a83e198  fffff800`016c88cf nt!KiDispatchException+0×16f
fffff880`0a83e1a0  fffff880`0a83e968
fffff880`0a83e1a8  fffff880`0a83e1d0
fffff880`0a83e1b0  fffff880`00000000
fffff880`0a83e1b8  00000000`00000000
fffff880`0a83e1c0  00000000`00000000
fffff880`0a83e1c8  00000000`00000000
[…]

0: kd> .cxr fffff880`0a83e1d0
rax=0000000000000009 rbx=fffffa800d4c1de0 rcx=0000000000000000
rdx=fffff8800a83ece0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff800016ad74f rsp=fffff8800a83eba0 rbp=00000000a000000c
r8=fffff8800a83ecd8  r9=fffff8800a83ecc0 r10=0000000000000000
r11=fffff8800a83ed58 r12=0000000000000000 r13=0000000000000000
r14=fffffa800d4bf9c0 r15=fffffa800d4c1ea0
iopl=0  nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b  efl=00010246
nt!IopCompleteRequest+0x12f:
fffff800`016ad74f 48894108 mov qword ptr [rcx+8],rax ds:002b:00000000`00000008=????????????????

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:

The First Comprehensive Software Diagnostics Service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Activity Resonance pattern is observed when two products from different vendors compete in some functional domain such malware detection. In the example below ApplicationA and AVDriverA modules belong to Vendor A and AV-B module belongs to Vendor B. Both threads are spiking threads blocking all other activity in the system:

0: kd> !running

System Processors: (0000000000000003)
Idle Processors: (0000000000000000) (0000000000000000) (0000000000000000) (0000000000000000)

Prcbs             Current           Next
0    fffff80001845e80  fffffa8004350060                    ................
1    fffff880009c4180  fffffa80028e7060                    ................

0: kd> !thread fffffa8004350060 ff
THREAD fffffa8004350060  Cid 14424.14b34  Teb: 000000007efdb000 Win32Thread: fffff900c1d32c30 RUNNING on processor 0
Not impersonating
DeviceMap                 fffff8a00148fe80
Owning Process            fffffa8003d6cb30       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      10568630       Ticks: 0
Context Switch Count      345                 LargeStack
UserTime                  00:02:21.360
KernelTime                01:09:32.130
Win32 Start Address ApplicationA!mainCRTStartup (0×0000000000404c1b)
Stack Init fffff88006c71db0 Current fffff88006c71670
Base fffff88006c72000 Limit fffff88006c6a000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`06c70ec0 fffff880`0197d53c AVDriverA+0×15d69
fffff880`06c70f10 fffff880`01988556 AVDriverA+0×1453c
fffff880`06c70fd0 fffff880`019886a8 AVDriverA+0×1f556
fffff880`06c71000 fffff800`0198ebfd AVDriverA+0×1f6a8
fffff880`06c71060 fffff800`019bf4f2 nt! ?? ::NNGAKEGL::`string’+0×2a6fd
fffff880`06c711e0 fffff800`019c3385 nt!PspCreateThread+0×246
fffff880`06c71460 fffff800`016d28d3 nt!NtCreateThreadEx+0×25d
fffff880`06c71bb0 00000000`76e61d9a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`06c71c20)
00000000`0008e178 00000000`74990411 ntdll!ZwCreateThreadEx+0xa
00000000`0008e180 00000000`7497cf87 wow64!whNtCreateThreadEx+0×815
00000000`0008e350 00000000`748c2776 wow64!Wow64SystemServiceEx+0xd7
00000000`0008ec10 00000000`7497d07e wow64cpu!TurboDispatchJumpAddressEnd+0×2d
00000000`0008ecd0 00000000`7497c549 wow64!RunCpuSimulation+0xa
00000000`0008ed20 00000000`76e54956 wow64!Wow64LdrpInitialize+0×429
00000000`0008f270 00000000`76e51a17 ntdll!LdrpInitializeProcess+0×17e4
00000000`0008f760 00000000`76e3c32e ntdll! ?? ::FNODOBFM::`string’+0×29220
00000000`0008f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

0: kd> !thread fffffa80028e7060 ff
THREAD fffffa80028e7060  Cid 0dc4.0e5c  Teb: 000000007efa4000 Win32Thread: 0000000000000000 RUNNING on processor 1
Not impersonating
DeviceMap                 fffff8a000008b30
Owning Process            fffffa8002817060       Image:         AV-B.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      10568617       Ticks: 13 (0:00:00:00.203)
Context Switch Count      1763138
UserTime                  00:04:26.765
KernelTime                03:09:31.140
Win32 Start Address AV-B (0×00000000004289f2)
Stack Init fffff88003b88db0 Current fffff88003b88900
Base fffff88003b89000 Limit fffff88003b83000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`03b88660 fffff800`019919a9 nt!ObReferenceObjectSafe+0xf
fffff880`03b88690 fffff800`01991201 nt!PsGetNextProcess+0×81
fffff880`03b886e0 fffff800`019dcef6 nt!ExpGetProcessInformation+0×774
fffff880`03b88830 fffff800`019dd949 nt!ExpQuerySystemInformation+0xfb4
fffff880`03b88be0 fffff800`016d28d3 nt!NtQuerySystemInformation+0×4d
fffff880`03b88c20 00000000`76e6167a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`03b88c20)
00000000`0118e708 00000000`74987da7 ntdll!NtQuerySystemInformation+0xa
00000000`0118e710 00000000`74988636 wow64!whNT32QuerySystemProcessInformationEx+0×93
00000000`0118e760 00000000`7498a0e9 wow64!whNtQuerySystemInformation_SpecialQueryCase+0×466
00000000`0118e800 00000000`7497cf87 wow64!whNtQuerySystemInformation+0xf1
00000000`0118e840 00000000`748c2776 wow64!Wow64SystemServiceEx+0xd7
00000000`0118f100 00000000`7497d07e wow64cpu!TurboDispatchJumpAddressEnd+0×2d
00000000`0118f1c0 00000000`7497c549 wow64!RunCpuSimulation+0xa
00000000`0118f210 00000000`76e8e707 wow64!Wow64LdrpInitialize+0×429
00000000`0118f760 00000000`76e3c32e ntdll! ?? ::FNODOBFM::`string’+0×29364
00000000`0118f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

• Title: Memory Dump Analysis Anthology, Volume 1
• Author: Dmitry Vostokov
• Publisher: OpenTask (Summer 2012)
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 800 pages
• ISBN-13: 978-1-908043-35-1

• Hardcover: 800 pages
• ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to many questions on recommended books to learn assembly language asked during Accelerated Windows Memory Dump Analysis training sessions we provide these references:

Windows Debugging: Practical Foundations
x64 Windows Debugging: Practical Foundations

Each book can be read independently although some platform-independent content overlaps. x64 bit book focuses on 64-bit only.

We believe these books provide all necessary motivation, context and practical foundation for other in-depth assembly language textbooks on the market.

I’m also working on the similar book for x64 Mac OS X.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Address space-wide search for errors and status codes may show Coincidental Error Code pattern:

0:000> !heap -x -v c0000005
Search VM for address range c0000005 - c0000005 : 028690b8 (c0000005), [...]

0:000> dd 028690b8 l1
028690b8  c0000005

In such cases we need to check whether the addresses belong to volatile regions such as stack because it is possible to have such values as legitimate code and image data:

0:000> !address 028690b8
Usage:                  Image
Allocation Base:        02700000
Base Address:           02869000
End Address:            02874000
Region Size:            0000b000
Type:                   01000000 MEM_IMAGE
State:                  00001000 MEM_COMMIT
Protect:                00000002 PAGE_READONLY
More info:              lmv m ModuleA
More info:              !lmi ModuleA
More info:              ln 0×28690b8

0:000> u 028690b8
ModuleA!ComputeB:
028690b8 050000c000      add     eax,0C00000h
[...]

Another example:

0:000> !heap -x -v c0000005
Search VM for address range 00000000c0000005 - 00000000c0000005 : 7feff63ab60 (c0000005),

0:000> !address 7feff63ab60
Usage:                  Image
Allocation Base:        000007fe`ff460000
Base Address:           000007fe`ff635000
End Address:            000007fe`ff63c000
Region Size:            00000000`00007000
Type:                   01000000 MEM_IMAGE
State:                  00001000 MEM_COMMIT
Protect:                00000004 PAGE_READWRITE
More info:              lmv m ole32
More info:              !lmi ole32
More info:              ln 0×7feff63ab60

0:000> dp 7feff63ab60
000007fe`ff63ab60  00000000`c0000005 c0000194`00000001
000007fe`ff63ab70  00000001`00000000 00000000`c00000aa
000007fe`ff63ab80  80000002`00000001 00000001`00000000
000007fe`ff63ab90  00000000`c0000096 c000001d`00000001
000007fe`ff63aba0  00000001`00000000 00000000`80000003
000007fe`ff63abb0  c00000fd`00000001 00000001`00000000
000007fe`ff63abc0  00000000`c0000235 c0000006`00000001
000007fe`ff63abd0  00000001`00000000 00000000`c0000420

In the latter case the data structure suggests a table of errors:

0:000> ln 7feff63ab60
(000007fe`ff63ab60)   ole32!gReportedExceptions

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was recently asked by a group of trainees to outline a simple approach to proceed after opening a memory dump. So I came up with these 7 steps:

1. !analyze -v [-hang]
2. Exception (Bugcheck): stack trace analysis with d* and lmv
3. !locks
4. !runaway f (!running)
5. Dump all (processes and) thread stack traces [with 32-bit] ~*kv (!process 0 ff)
6. Search for signs/patterns of abnormal behavior (exceptions, wait chains, message boxes [, from your custom checklist])
7. Narrow analysis down to a specific thread and dump raw stack data if needed [repeat*]

(commands/options in brackets denote kernel/complete dump variation)
[notes in square brackets denote additional options, such as x64 specifics, your product details, etc.]

What are your steps? I would be interested to hear about alternative analysis steps, techniques, etc.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of attendees of accelerated memory dump analysis training pointed me to the possible effects of -g option for AeDebug custom postmortem debugger command line for CDB, NTSD or WinDbg. So I tested that with x64 TestWER tool (should be the same with x86 version) and indeed there are differences.

With -g option with have this stack trace:

AeDebug\Debugger = "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -p %ld -e %ld -g

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012f210 00000001`40004148 TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0x7e
00000000`0012f250 00000001`40004388 TestWER64!_AfxDispatchCmdMsg+0xc4
00000000`0012f280 00000001`40003552 TestWER64!CCmdTarget::OnCmdMsg+0x180
00000000`0012f2e0 00000001`4000cc44 TestWER64!CDialog::OnCmdMsg+0x32
00000000`0012f320 00000001`4000d877 TestWER64!CWnd::OnCommand+0xcc
00000000`0012f3b0 00000001`40008c2c TestWER64!CWnd::OnWndMsg+0x5f
00000000`0012f4f0 00000001`4000c272 TestWER64!CWnd::WindowProc+0x38
00000000`0012f530 00000001`4000c32d TestWER64!AfxCallWndProc+0xfe
00000000`0012f5d0 00000000`77519bd1 TestWER64!AfxWndProc+0x59
00000000`0012f610 00000000`77516aa8 USER32!UserCallWinProcCheckWow+0x1ad
00000000`0012f6d0 00000000`77516bad USER32!SendMessageWorker+0x682
00000000`0012f760 000007fe`fccb0bbf USER32!SendMessageW+0x5c
00000000`0012f7b0 000007fe`fccb47df COMCTL32!Button_ReleaseCapture+0x157
00000000`0012f7f0 00000000`77519bd1 COMCTL32!Button_WndProc+0xcbf
00000000`0012f8b0 00000000`775198da USER32!UserCallWinProcCheckWow+0x1ad
00000000`0012f970 00000000`775167c2 USER32!DispatchMessageWorker+0x3b5
00000000`0012f9f0 00000001`400079cc USER32!IsDialogMessageW+0x153
00000000`0012fa80 00000001`40009148 TestWER64!CWnd::IsDialogMessageW+0x38
00000000`0012fab0 00000001`40003513 TestWER64!CWnd::PreTranslateInput+0x28
00000000`0012fae0 00000001`4000b696 TestWER64!CDialog::PreTranslateMessage+0xc3
00000000`0012fb10 00000001`40004c1f TestWER64!CWnd::WalkPreTranslateTree+0x3a
00000000`0012fb40 00000001`40004c7f TestWER64!AfxInternalPreTranslateMessage+0x67
00000000`0012fb70 00000001`40004e26 TestWER64!AfxPreTranslateMessage+0x23
00000000`0012fba0 00000001`40004e6b TestWER64!AfxInternalPumpMessage+0x3a
00000000`0012fbd0 00000001`4000aba6 TestWER64!AfxPumpMessage+0x1b
00000000`0012fc00 00000001`40003e4a TestWER64!CWnd::RunModalLoop+0xea
00000000`0012fc60 00000001`40024da4 TestWER64!CDialog::DoModal+0x1c6
00000000`0012fd10 00000001`40024625 TestWER64!CTestDefaultDebuggerApp::InitInstance+0xc4
00000000`0012fe70 00000001`400153c2 TestWER64!AfxWinMain+0x75
00000000`0012feb0 00000000`77ad652d TestWER64!__tmainCRTStartup+0x186
00000000`0012ff60 00000000`77c0c521 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

0:000> r
rax=0000000000000000 rbx=0000000000000001 rcx=000000000012fd50
rdx=00000000000003e8 rsi=000000000012fd50 rdi=000000014002daa0
rip=00000001400247ae rsp=000000000012f210 rbp=0000000000000111
r8=0000000000000000  r9=0000000140024730 r10=0000000140024730
r11=000000000012f310 r12=0000000000000000 r13=00000000000003e8
r14=0000000000000110 r15=0000000000000001
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244
TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0x7e:
00000001`400247ae c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=????????

Without -g option we also see exception processing (highlighted in red):

AeDebugger\Debugger = "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -p %ld -e %ld

0:000> kL
Child-SP RetAddr Call Site
00000000`0012e368 000007fe`fe301420 ntdll!ZwWaitForMultipleObjects+0xa
00000000`0012e370 00000000`77ae2cf3 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`0012e470 00000000`77b590f5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`0012e500 00000000`77b59277 kernel32!WerpReportFaultInternal+0×215
00000000`0012e5a0 00000000`77b592cf kernel32!WerpReportFault+0×77
00000000`0012e5d0 00000000`77b594ec kernel32!BasepReportFault+0×1f
00000000`0012e600 00000000`77c743b8 kernel32!UnhandledExceptionFilter+0×1fc
00000000`0012e6e0 00000000`77bf85a8 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`0012e710 00000000`77c09d0d ntdll!_C_specific_handler+0×8c
00000000`0012e780 00000000`77bf91af ntdll!RtlpExecuteHandlerForException+0xd
00000000`0012e7b0 00000000`77c31278 ntdll!RtlDispatchException+0×45a
00000000`0012ee90 00000001`400247ae ntdll!KiUserExceptionDispatcher+0×2e
00000000`0012f450 00000001`40004148 TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0×7e
00000000`0012f490 00000001`40004388 TestWER64!_AfxDispatchCmdMsg+0xc4
00000000`0012f4c0 00000001`40003552 TestWER64!CCmdTarget::OnCmdMsg+0×180
00000000`0012f520 00000001`4000cc44 TestWER64!CDialog::OnCmdMsg+0×32
00000000`0012f560 00000001`4000d877 TestWER64!CWnd::OnCommand+0xcc
00000000`0012f5f0 00000001`40008c2c TestWER64!CWnd::OnWndMsg+0×5f
00000000`0012f730 00000001`4000c272 TestWER64!CWnd::WindowProc+0×38
00000000`0012f770 00000001`4000c32d TestWER64!AfxCallWndProc+0xfe
00000000`0012f810 00000000`77519bd1 TestWER64!AfxWndProc+0×59
00000000`0012f850 00000000`77516aa8 USER32!UserCallWinProcCheckWow+0×1ad
00000000`0012f910 00000000`77516bad USER32!SendMessageWorker+0×682
00000000`0012f9a0 00000000`7751eda7 USER32!SendMessageW+0×5c
00000000`0012f9f0 00000001`400079cc USER32!IsDialogMessageW+0×85f
00000000`0012fa80 00000001`40009148 TestWER64!CWnd::IsDialogMessageW+0×38
00000000`0012fab0 00000001`40003513 TestWER64!CWnd::PreTranslateInput+0×28
00000000`0012fae0 00000001`4000b696 TestWER64!CDialog::PreTranslateMessage+0xc3
00000000`0012fb10 00000001`40004c1f TestWER64!CWnd::WalkPreTranslateTree+0×3a
00000000`0012fb40 00000001`40004c7f TestWER64!AfxInternalPreTranslateMessage+0×67
00000000`0012fb70 00000001`40004e26 TestWER64!AfxPreTranslateMessage+0×23
00000000`0012fba0 00000001`40004e6b TestWER64!AfxInternalPumpMessage+0×3a
00000000`0012fbd0 00000001`4000aba6 TestWER64!AfxPumpMessage+0×1b
00000000`0012fc00 00000001`40003e4a TestWER64!CWnd::RunModalLoop+0xea
00000000`0012fc60 00000001`40024da4 TestWER64!CDialog::DoModal+0×1c6
00000000`0012fd10 00000001`40024625 TestWER64!CTestDefaultDebuggerApp::InitInstance+0xc4
00000000`0012fe70 00000001`400153c2 TestWER64!AfxWinMain+0×75
00000000`0012feb0 00000000`77ad652d TestWER64!__tmainCRTStartup+0×186
00000000`0012ff60 00000000`77c0c521 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

I now prefer omitting -g option to get stack traces equivalent to manual crash dumps saved by userdump.exe on pre-Vista platforms and Task Manager on later platforms.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Advanced training sessions time may not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

• Title: Advanced Windows Memory Dump Analysis with Data Structures: Training Course Transcript and WinDbg Practice Exercises with Notes
• Description: The full transcript of Memory Dump Analysis Services Training with 10 step-by-step exercises, notes, and selected Q&A.
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Publisher: OpenTask (January 2012)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 180 pages
• ISBN-13: 978-1908043344

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have memory leaks related to the growing number of page tables. One reason for that could be the growing number of zombie processes (noticeable with tens of thousands of them).

1: kd> !process 0 0
[...] 
PROCESS fffffa80266bd6f0
    SessionId: 0  Cid: 0a6c    Peb: 7fffffdc000  ParentCid: 03ac
    DirBase: 9d35a000  ObjectTable: fffff8a00170ac80  HandleCount: 152.
    Image: svchost.exe
[…] 
PROCESS fffffa8027de9b30
    SessionId: 0  Cid: 21d0    Peb: 7fffffdf000  ParentCid: 02e0
    DirBase: 37881000  ObjectTable: 00000000  HandleCount:   0.
    Image: conhost.exe
[…] 
PROCESS fffffa8028eb0600
    SessionId: 0  Cid: ab88    Peb: 7fffffdf000  ParentCid: 02e0
    DirBase: 27a2f000  ObjectTable: 00000000  HandleCount:   0.
    Image: conhost.exe
[…]

Even zombies have at least one remaining page (page directory) from the former page tables of their virtual to physical memory mapping (!dd is the same as dd command but for physical memory):

1: kd> !dd 9d35a000
#9d35a000 9dd62867 03c00000 00000000 00000000
#9d35a010 00000000 00000000 00000000 00000000
#9d35a020 00000000 00000000 00000000 00000000
#9d35a030 00000000 00000000 00000000 00000000
#9d35a040 00000000 00000000 00000000 00000000
#9d35a050 00000000 00000000 00000000 00000000
#9d35a060 00000000 00000000 00000000 00000000
#9d35a070 00000000 00000000 9d45e867 49500000

1: kd> !dd 37881000
#37881000 00000000 00000000 00000000 00000000
#37881010 00000000 00000000 00000000 00000000
#37881020 00000000 00000000 00000000 00000000
#37881030 00000000 00000000 00000000 00000000
#37881040 00000000 00000000 00000000 00000000
#37881050 00000000 00000000 00000000 00000000
#37881060 00000000 00000000 00000000 00000000
#37881070 00000000 00000000 00000000 00000000

1: kd> !dd 27a2f000
#27a2f000 00000000 00000000 00000000 00000000
#27a2f010 00000000 00000000 00000000 00000000
#27a2f020 00000000 00000000 00000000 00000000
#27a2f030 00000000 00000000 00000000 00000000
#27a2f040 00000000 00000000 00000000 00000000
#27a2f050 00000000 00000000 00000000 00000000
#27a2f060 00000000 00000000 00000000 00000000
#27a2f070 00000000 00000000 00000000 00000000

We also see that 2 conhost.exe processes have identical physical to virtual mapping because their user space mappings are no longer valid (zeroed) and svchost.exe process has user space mapping (in blue italics):

1: kd> !ptov 27a2f000
Amd64PtoV: pagedir 27a2f000
27a2f000 fffff6fb`7dbed000
71530000 fffff6fb`7dbee000
19d000 fffff6fb`7dbef000
199000 fffff6fb`7dbf0000
b6a04000 fffff6fb`7dbf1000
b1f57000 fffff6fb`7dbf2000
29c4000 fffff6fb`7dbf3000
1c53000 fffff6fb`7dbf5000 
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

1: kd> !ptov 37881000
Amd64PtoV: pagedir 37881000
37881000 fffff6fb`7dbed000
8d482000 fffff6fb`7dbee000
19d000 fffff6fb`7dbef000
199000 fffff6fb`7dbf0000
b6a04000 fffff6fb`7dbf1000
b1f57000 fffff6fb`7dbf2000
29c4000 fffff6fb`7dbf3000
1c53000 fffff6fb`7dbf5000
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

1: kd> !ptov 9d35a000
Amd64PtoV: pagedir 9d35a000
9e587000 10000
6871e000 20000
af5aa000 30000
af5ab000 31000
afaac000 32000
afbad000 33000
af2f5000 40000
9d66b000 50000
22199000 60000
9d962000 e5000
9d261000 e6000
9dc60000 e7000
9d256000 ea000
9d84f000 eb000
9e4ec000 ec000
9e081000 ed000
9d876000 ee000
9e271000 ef000
b8bfd000 f0000
b8efe000 f1000
b86ff000 f2000
b5302000 f3000
b5202000 f4000
b5502000 f5000
b7f03000 f6000
b8404000 f7000
b8415000 100000
b8b16000 101000
b1b17000 102000
[…]
2cd4000 77512000
5d7000 77515000
5d8000 77516000
4d9000 77517000
b358f000 77590000
aef04000 77591000
68624000 77592000
64b26000 77593000
af4c6000 77595000
b2042000 7efe0000
b2143000 7efe1000
b1a56000 7efe2000
b1a57000 7efe3000
b1b58000 7efe4000
1ba000 7ffe0000
9da69000 bfeb0000
aeeae000 ffea0000
af191000 ffea1000
9d76a000 ffea2000
ae793000 ffea3000
9dc8e000 ffea5000
b7eb7000 ffea6000
9dffc000 ffea7000
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

In order to check user space virtual addresses we have to switch to the corresponding process context:

1: kd> !pte fffffa80`28f2d000
                                           VA fffffa8028f2d000
PXE at FFFFF6FB7DBEDFA8    PPE at FFFFF6FB7DBF5000    PDE at FFFFF6FB7EA00A38    PTE at FFFFF6FD40147968
contains 0000000001C53863  contains 0000000001C54863  contains 0000000049320863  contains 000000002E4D8963
pfn 1c53      —DA–KWEV  pfn 1c54      —DA–KWEV  pfn 49320     —DA–KWEV  pfn 2e4d8     -G-DA–KWEV

1: kd> .process /r /p fffffa80266bd6f0
Implicit process is now fffffa80`266bd6f0
Loading User Symbols

1: kd> !pte 10000
                                           VA 0000000000010000
PXE at FFFFF6FB7DBED000    PPE at FFFFF6FB7DA00000    PDE at FFFFF6FB40000000    PTE at FFFFF68000000080
contains 03C000009DD62867  contains 031000009D865867  contains 7C2000009DD66867  contains 9CB000009E587867
pfn 9dd62     —DA–UWEV  pfn 9d865     —DA–UWEV  pfn 9dd66     —DA–UWEV  pfn 9e587     —DA–UW-V

This pattern came to our attention after several customers complained about the growing number of memory allocated for page tables which exceeded a gigabyte after several days.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -