Software Diagnostics Library

Sometimes we have a value or a pointer or a handle and would like to know all memory addresses that reference it. This can be done by virtual memory search (s WinDbg command). If you look for references in code (for example, or pool tags please see this case study) you can combine search with !for_each_module WinDbg extension command. There is also !search command for physical pages. We cover this Value References pattern in the forthcoming Advanced Windows Memory Dump Analysis training with a step-by-step complete memory dump analysis exercise. For object references there is also recently added !obtrace command with good examples in WinDbg help.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Certain System Objects can be found in object directory and can be useful to see additional system and other product activity. For example, in a complete memory dump from Accelerated .NET Memory Dump Analysis training we see that LowCommitCondition event is signalled:

1: kd> !object \KernelObjects
Object: 85a08030  Type: (82b38ed0) Directory
    ObjectHeader: 85a08018 (old version)
    HandleCount: 0  PointerCount: 19
    Directory Object: 85a074c0  Name: KernelObjects

    Hash Address  Type          Name
    ---- -------  ----          ----
     02  82b7b0b8 Event         HighCommitCondition
     04  82b7b780 Event         HighMemoryCondition
     10  82b7b178 Event         LowNonPagedPoolCondition
     11  82b7b138 Event         HighNonPagedPoolCondition
     17  82b7b0f8 Event         LowCommitCondition
     20  82b78d08 Event         SuperfetchParametersChanged
         82b6eb58 Event         BootLoaderTraceReady
     23  84bfdd58 Session       Session0
         82b78c88 Event         PrefetchTracesReady
     24  84b7d1f8 Session       Session1
     25  82b78cc8 Event         SuperfetchScenarioNotify
         82b7b740 Event         LowPagedPoolCondition
     26  82b7b1b8 Event         HighPagedPoolCondition
         82b7a030 Event         MemoryErrors
     28  82b78c48 Event         SuperfetchTracesReady
     32  82b7b7c0 Event         LowMemoryCondition
         85a09d00 KeyedEvent    CritSecOutOfMemoryEvent
     34  82b7b078 Event         MaximumCommitCondition

1: kd> dt _DISPATCHER_HEADER 82b7b0f8
ntdll!_DISPATCHER_HEADER
   +0x000 Type             : 0 ''
   +0x001 Abandoned        : 0 ''
   +0x001 Absolute         : 0 ''
   +0x001 NpxIrql          : 0 ''
   +0x001 Signalling       : 0 ''
   +0x002 Size             : 0x4 ''
   +0x002 Hand             : 0x4 ''
   +0x003 Inserted         : 0 ''
   +0x003 DebugActive      : 0 ''
   +0x003 DpcActive        : 0 ''
   +0x000 Lock             : 0n262144
   +0×004 SignalState      : 0n1
   +0×008 WaitListHead     : _LIST_ENTRY [ 0×82b7b100 - 0×82b7b100 ]

If we check virtual memory statistics we see lots of free space for the currrent physical memory and pagefile: 

1: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:      261872 (   1047488 Kb)
 Page File: \??\C:\pagefile.sys
   Current:   1354688 Kb  Free Space:     53120 Kb
   Minimum:   1354688 Kb  Maximum:      4194304 Kb
 Available Pages:      180984 (    723936 Kb)
 ResAvail Pages:       216475 (    865900 Kb)
 Locked IO Pages:           0 (         0 Kb)
 Free System PTEs:     352925 (   1411700 Kb)
 Modified Pages:          129 (       516 Kb)
 Modified PF Pages:        94 (       376 Kb)
 NonPagedPool Usage:        0 (         0 Kb)
 NonPagedPoolNx Usage:  16894 (     67576 Kb)
 NonPagedPool Max:     192350 (    769400 Kb)
 PagedPool 0 Usage:      5957 (     23828 Kb)
 PagedPool 1 Usage:      3218 (     12872 Kb)
 PagedPool 2 Usage:       965 (      3860 Kb)
 PagedPool 3 Usage:      1311 (      5244 Kb)
 PagedPool 4 Usage:      1064 (      4256 Kb)
 PagedPool Usage:       12515 (     50060 Kb)
 PagedPool Maximum:    523264 (   2093056 Kb)
 Session Commit:         5021 (     20084 Kb)
 Shared Commit:         15023 (     60092 Kb)
 Special Pool:              0 (         0 Kb)
 Shared Process:         1938 (      7752 Kb)
 PagedPool Commit:      12523 (     50092 Kb)
 Driver Commit:          2592 (     10368 Kb)
 Committed pages:      402494 (   1609976 Kb)
 Commit limit:         589254 (   2357016 Kb)
[...]

Another example is from Windows 7 memory dump I used for Fundamentals of Complete Crash and Hang Memory Dump Analysis presentation. Here we can find WER reporting mutant in session 1 object directory and get problem PID from its name:

0: kd> !object \Sessions\1\BaseNamedObjects\
Object: fffff8a0016eb290  Type: (fffffa800426df30) Directory
    ObjectHeader: fffff8a0016eb260 (new version)
    HandleCount: 57  PointerCount: 217
    Directory Object: fffff8a0016e9220  Name: BaseNamedObjects

    Hash Address          Type          Name
    ---- -------          ----          ----
     00  fffffa8008437670 Event         STOP_HOOKING64
[...]
    08  fffffa80044baa40 Mutant        WERReportingForProcess1788
[…]

0: kd> !process 0n1788 1
Searching for Process with Cid == 6fc
Cid handle table at fffff8a00180b000 with 21248 entries in use

PROCESS fffffa8004364060
    SessionId: 1  Cid: 06fc    Peb: 7fffffd4000  ParentCid: 0840
    DirBase: 5fbc2000  ObjectTable: fffff8a004c8e930  HandleCount:  16.
    Image: ApplicationD.exe
    VadRoot fffffa8009d85170 Vads 34 Clone 0 Private 206. Modified 0. Locked 0.
    DeviceMap fffff8a001ce6b90
    Token                             fffff8a003eab060
    ElapsedTime                       00:01:51.543
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (483, 50, 345) (1932KB, 200KB, 1380KB)
    PeakWorkingSetSize                483
    VirtualSize                       13 Mb
    PeakVirtualSize                   13 Mb
    PageFaultCount                    481
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      231

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to functions we also have module variables like nt!MmPagedPoolCommit in Windows 7:

0: kd> x nt!MmPagedPool*
fffff800`031148d0 nt!MmPagedPoolInfo = <no type information>
fffff800`03092d20 nt!MmPagedPoolCommit = <no type information>
fffff800`031141a0 nt!MmPagedPoolEnd = <no type information>
fffff800`031175c0 nt!MmPagedPoolWs = <no type information>

If we are not sure whether we have a function or Module Variable we can try to disassemble:

0: kd> u nt!MmPagedPoolCommit
nt!MmPagedPoolCommit:
fffff800`03092d20 e3b2            jrcxz   nt!MmTotalNonPagedPoolQuota+0x4 (fffff800`03092cd4)
fffff800`03092d22 0000            add     byte ptr [rax],al
fffff800`03092d24 0000            add     byte ptr [rax],al
fffff800`03092d26 0000            add     byte ptr [rax],al
fffff800`03092d28 0000            add     byte ptr [rax],al
fffff800`03092d2a 0000            add     byte ptr [rax],al
fffff800`03092d2c 0000            add     byte ptr [rax],al
fffff800`03092d2e 0000            add     byte ptr [rax],al

Here the value is probably in pages so we multiply by 4 to get Kb value and compare to the output of !vm command:

0: kd> dp nt!MmPagedPoolCommit
fffff800`03092d20  00000000`0000b2e3 00000000`00000000
fffff800`03092d30  00000000`00000000 00000000`00000000
fffff800`03092d40  00000000`00000001 00000000`00000000
fffff800`03092d50  00000000`00000000 00000000`00060107
fffff800`03092d60  fffff800`03092d60 fffff800`03092d60
fffff800`03092d70  00000000`00000000 00000000`0001e972
fffff800`03092d80  fffff900`c0000000 00000000`00000002
fffff800`03092d90  fffff880`071dc0a8 fffff880`057340a8

0: kd> ? b2e3 * 4
Evaluate expression: 183180 = 00000000`0002cb8c

0: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:     1035228 (   4140912 Kb)
 Page File: \??\C:\pagefile.sys
   Current:   4448112 Kb  Free Space:   4448108 Kb
   Minimum:   4448112 Kb  Maximum:     12422736 Kb
Unimplemented error for MiSystemVaTypeCount
 Available Pages:      594029 (   2376116 Kb)
 ResAvail Pages:       889795 (   3559180 Kb)
 Locked IO Pages:           0 (         0 Kb)
 Free System PTEs:   33556870 ( 134227480 Kb)
 Modified Pages:        20079 (     80316 Kb)
 Modified PF Pages:     19441 (     77764 Kb)
 NonPagedPool Usage: 50865104 ( 203460416 Kb)
 NonPagedPoolNx Usage:  28163 (    112652 Kb)
 NonPagedPool Max:     763396 (   3053584 Kb)
 ********** Excessive NonPaged Pool Usage *****
 PagedPool 0 Usage:     39420 (    157680 Kb)
 PagedPool 1 Usage:      5194 (     20776 Kb)
 PagedPool 2 Usage:       367 (      1468 Kb)
 PagedPool 3 Usage:       338 (      1352 Kb)
 PagedPool 4 Usage:       440 (      1760 Kb)
 PagedPool Usage:       45759 (    183036 Kb)
 PagedPool Maximum:  33554432 ( 134217728 Kb)
 Session Commit:         8112 (     32448 Kb)
 Shared Commit:         31802 (    127208 Kb)
 Special Pool:              0 (         0 Kb)
 Shared Process:        10765 (     43060 Kb)
 PagedPool Commit:      45795 (    183180 Kb)
 Driver Commit:         13773 (     55092 Kb)
 Committed pages:      540998 (   2163992 Kb)
 Commit limit:        2146794 (   8587176 Kb)
[…]

Knowledge of available module variables is useful because some of them are not included in WinDbg extension command output. For their list please consult Windows Internals book. Also useful variables can be found in other modules as well, for example, srv!srvcomputername:

0: kd> dS srv!srvcomputername
fffff8a0`0344b090  "MYNOTEBOOK"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we need to narrow general stack trace collection to a few threads that satisfy some predicate, for example, all threads with kernel time spent greater than some value or all suspended threads or all threads that wait for a specific synchronization object type. We call this pattern variant Stack Trace Collection (predicate). This can be implemented using WinDbg scripts and / or debugger extensions.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If not for you then definitely for me because I now diagnose Spiking Thread pattern much faster. One of the readers if this blog asked me whether there is !runaway command equivalent for kernel and complete memory dumps. So, after some thinking I gave it a try especially in the context of WinDbg scripting exercises designed for Advanced Windows Memory Dump Analysis training. As a result I wrote 2 scripts initially that you can try yourself. Their output here is taken from a complete memory dump I used for Fundamentals of Complete Crash and Hang Memory Dump Analysis presentation.

The first one dumps the most CPU consuming threads for user and kernel mode:

$$
$$ krunawaymost.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest UserTime value: ”
? @$t0
!thread @$t2 ff
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest KernelTime value: ”
? @$t0
!thread @$t2 ff

0: kd> $$><c:\Scripts\krunawaymost.wds
The largest UserTime value:
Evaluate expression: 5470 = 00000000`0000155e
THREAD fffffa800451d720  Cid 1418.17fc  Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa800442ab30       Image:         ApplicationE.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22295          Ticks: 0
Context Switch Count      27960            
UserTime                  00:01:25.332
KernelTime                00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3

The largest KernelTime value:
Evaluate expression: 187 = 00000000`000000bb
THREAD fffffa80098d7b60  Cid 07bc.0a14  Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
    fffffa8008a4a030  NotificationEvent
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa80096beb30       Image:         dwm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22294          Ticks: 1 (0:00:00:00.015)
Context Switch Count      15473                 LargeStack
UserTime                  00:00:06.801
KernelTime                00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The second script takes two arguments and shows all threads that have UserTime and KernelTime ticks value greater than (you can have the idea of the maximum from the previous script):

$$
$$ krunawaygt.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); r $t0 = $arg1; .if (@$t1 > @$t0) {!thread @#Thread ff}”
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); r $t0 = $arg2; .if (@$t1 > @$t0) {!thread @#Thread ff}”

Using hints from the previous script run (the largest UserTime ticks value is 0×155e) we now get threads that spent more than 0×100 ticks in user mode:

0: kd> $$>a<c:\Scripts\krunawaygt.wds 100 100
THREAD fffffa800843e060  Cid 03f4.0658  Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa800843c2c0  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      21211          Ticks: 1084 (0:00:00:16.910)
Context Switch Count      6028            
UserTime                  00:00:10.140
KernelTime                00:00:00.296
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d4bc70 Current fffff88009d4b660
Base fffff88009d4c000 Limit fffff88009d46000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mprtp.dll -
Child-SP          RetAddr           Call Site
fffff880`09d4b6a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d4b7e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d4b870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d4b930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d4b9c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d4ba70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d4bae0)
00000000`0209fb08 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`0209fb10 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`0209fb70 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`0209fbb0 00000000`0045cbc0 mprtp!MpPluginSignatureChange+0×3e170
00000000`0209fbb8 000007fe`fbac25ff 0×45cbc0
00000000`0209fbc0 00000000`00466610 FLTLIB!FilterGetMessage+0×2b
00000000`0209fc20 00000000`00000000 0×466610

THREAD fffffa800845c060  Cid 03f4.065c  Teb: 000007fffff8e000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa800843c2c0  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      21520          Ticks: 775 (0:00:00:12.090)
Context Switch Count      4979            
UserTime                  00:00:04.149
KernelTime                00:00:00.156
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d52c70 Current fffff88009d52660
Base fffff88009d53000 Limit fffff88009d4d000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mprtp.dll -
Child-SP          RetAddr           Call Site
fffff880`09d526a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d527e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d52870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d52930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d529c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d52a70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d52ae0)
00000000`01ccf498 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`01ccf4a0 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`01ccf500 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`01ccf540 00000000`0045d030 mprtp!MpPluginSignatureChange+0×3e170
00000000`01ccf548 000007fe`fbac25ff 0×45d030
00000000`01ccf550 00000000`004666b0 FLTLIB!FilterGetMessage+0×2b
00000000`01ccf5b0 00000000`00000000 0×4666b0

THREAD fffffa80092b7060  Cid 03f4.1268  Teb: 000007fffff6a000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Alertable
    fffffa8009299140  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      7762           Ticks: 14533 (0:00:03:46.716)
Context Switch Count      3297            
UserTime                  00:00:06.489
KernelTime                00:00:00.499
Win32 Start Address ntdll!TppWorkerThread (0×00000000772bfbc0)
Stack Init fffff8800e620c70 Current fffff8800e620680
Base fffff8800e621000 Limit fffff8800e61b000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0e6206c0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0e620800 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`0e620890 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`0e620950 fffff800`02ecdb36 nt!IoRemoveIoCompletion+0×47
fffff880`0e6209e0 fffff800`02ee0ed3 nt!NtWaitForWorkViaWorkerFactory+0×285
fffff880`0e620ae0 00000000`772f2c1a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0e620ae0)
00000000`0540f998 00000000`772bfe0b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0540f9a0 00000000`7719652d ntdll!TppWorkerThread+0×2c9
00000000`0540fca0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0540fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa80098d7b60  Cid 07bc.0a14  Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
    fffffa8008a4a030  NotificationEvent
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa80096beb30       Image:         dwm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22294          Ticks: 1 (0:00:00:00.015)
Context Switch Count      15473                 LargeStack
UserTime                  00:00:06.801
KernelTime                00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa800451d720  Cid 1418.17fc  Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa800442ab30       Image:         ApplicationE.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22295          Ticks: 0
Context Switch Count      27960            
UserTime                  00:01:25.332
KernelTime                00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3

Memory Dump Analysis Services is now working to incorporate client-side WinDbg scripting into their CARE² architecture. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org - 

Does God play keyboard?

Dmitry Vostokov, Memoriarch, Founder of Memory Religion (Memorianity)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of exception related patterns is necessary, so I created this post:

• Multiple Exceptions (user mode)
• Multiple Exceptions (kernel mode)
• Multiple Exceptions (managed space)
• Multiple Exceptions (stowed)
• Hidden Exception (user space)
• Hidden Exception (kernel space)
• Hidden Exception (managed space)
• Managed Code Exception
• Custom Exception Handler (user space)
• Custom Exception Handler (kernel space)
• Nested Exceptions (unmanaged code)
• Nested Exceptions (managed code)
• C++ Exception
• Mixed Exception
• Exception Stack Trace
• Invalid Exception Information
• Problem Exception Handler
• Handled Exception (user space)
• Handled Exception (.NET CLR)
• Handled Exception (kernel space)
• FPU Exception
• Translated Exception
• Stored Exception
• Exception Module
• Software Exception
• Distributed Exception (managed code)
• Exception Reporting Thread
• COM Exception
• Saved Exception Context (.NET)
• Exception Collection
• Near Exception

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One pattern I missed is called FPU Exception and it sometimes happens where you least expect it. Here’s extract from one crash dump raw stack analysis showing exception context, record and the usage of r WinDbg command variant to display FPU registers:

0:002> dps 056c1000 057c0000  
[...]
057bdee0  00000008
057bdee4  00000000
057bdee8  057bed6c
057bdeec  0d6e3130
057bdef0  057c0000
057bdef4  057b9000
057bdef8  006e3138
057bdefc  057be200
057bdf00  7c90e48a ntdll!KiUserExceptionDispatcher+0xe
057bdf04  057bed6c
057bdf08  057bdf2c
057bdf0c  057bdf14
057bdf10  057bdf2c
057bdf14  c0000090
057bdf18  00000010
057bdf1c  00000000
057bdf20  79098cc0 mscorjit!Compiler::FlatFPIsSameAsFloat+0xd
057bdf24  00000001
057bdf28  00000000
057bdf2c  0001003f
057bdf30  00000000
057bdf34  00000000
057bdf38  00000000
057bdf3c  00000000
057bdf40  00000000
057bdf44  00000000
057bdf48  ffff1372
057bdf4c  fffffda1
057bdf50  ffffbfff 
[…]

0:002> .cxr 057bdf2c
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
mscorwks!SOTolerantBoundaryFilter+0x22:
79f5236b d9059823f579    fld     dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000

0:002> .exr 057bdf14
ExceptionAddress: 79098cc0 (mscorjit!Compiler::FlatFPIsSameAsFloat+0x0000000d)
   ExceptionCode: c0000090
  ExceptionFlags: 00000010
NumberParameters: 1
   Parameter[0]: 00000000

0:002> !error c0000090
Error code: (NTSTATUS) 0xc0000090 (3221225616) - {EXCEPTION}  Floating-point invalid operation.

0:002> rMF
Last set context:
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
fpcw=1372: rn 64 pu–d-  fpsw=FDA1: top=7 cc=1101 b-p—-i  fptw=BFFF
fopcode=045D  fpip=001b:79098cc0  fpdp=0023:057bea7c
st0=-1.#IND00000000000000000e+0000  st1= 0.006980626232475338220e-4916
st2= 6.543831490564206840810e-4932  st3=-0.003025663186207448300e+2614
st4= 2.000000000000000000000e+0000  st5= 6.291456000000000000000e+0006
st6= 1.000000000000000000000e+0000  st7= 2.500000000000000000000e-0001
mscorwks!SOTolerantBoundaryFilter+0×22:
79f5236b d9059823f579    fld     dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Frequently we observe that several trace messages form a single semantic unit we call Macrofunction where individual trace messages serve the role of microfunctions. We borrowed this idea and distinction from functionalist linguistics. An example would be a software trace fragment where messages log an attempt to update a database:

#     Module   PID  TID   Time         Message
[...]
42582 DBClient 5492 9476  11:04:33.398 Opening connection
[...]
42585 DBClient 5492 9476  11:04:33.398 Sending SQL command
[...]
42589 DBServer 6480 10288 11:04:33.399 Executing SQL command
[...]
42592 DBClient 5492 9476  11:04:33.400 Closing connection
[...]

Please note that these macrofunctions need not be from the same ATID in the traditional sense like in the example above unless we form adjoint threads from certain fragments like “DB”.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

LoL - Law of Large

Examples: Q. How did you resolve this support case? A. LoL number!

Explanation: The more support incidents you get, the larger their tracking numbers. So at some stage the law of large numbers comes into effect: there is always a similar incident in the past. Don’t confuse with LOL.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was recently asked to provide explanation on how to analyze (A)LPC wait chains and the output of !lpc and !alpc commands in kernel and complete memory dumps and so I compiled these patterns:

• LPC Deadlock
• LPC Wait Chain
• Paged Out Data
• Process Object Wait Chain (+ ALPC)
• Blocked Queue (ALPC)
• Semantic Structures (ALPC)
• RPC Wait Chain (+ how to distinguish between LPC and RPC)
• Screwbolt Wait Chain

LPC case studies:

http://www.dumpanalysis.org/blog/index.php/2008/12/19/blocked-gui-thread-wait-chain-and-virtualized-process-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2008/12/24/insufficient-memory-handle-leak-wait-chain-deadlock-inconsistent-dump-and-overaged-system-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/03/11/coupled-processes-wait-chains-message-box-waiting-thread-time-paged-out-data-incorrect-stack-trace-hidden-exception-unknown-component-and-execution-residue-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/06/09/inconsistent-dump-blocked-threads-wait-chains-incorrect-stack-trace-and-process-factory-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/08/11/stack-trace-collection-suspended-threads-not-my-version-special-process-main-thread-and-blocked-lpc-chain-threads-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/11/20/stack-trace-collection-missing-threads-waiting-time-critical-section-and-lpc-wait-chains-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/02/27/inconsistent-dump-stack-trace-collection-lpc-thread-process-executive-resource-wait-chains-missing-threads-and-waiting-thread-time-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/07/18/stack-trace-collection-special-process-lpc-and-critical-section-wait-chains-blocked-thread-coupled-machines-thread-waiting-time-and-irp-distribution-anomaly-pattern-cooperation/

ALPC case studies:

http://www.dumpanalysis.org/blog/index.php/2009/09/18/alpc-wait-chain-missing-threads-message-box-zombie-and-special-processes-pattern-cooperation/

RPC target:

http://www.dumpanalysis.org/blog/index.php/2008/07/11/in-search-of-lost-pid/

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

During the previous several months some companies and individuals expressed their interest in the training (the next one is scheduled for January) but its time was not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

• Title: Accelerated .NET Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
• Description: The full transcript of Memory Dump Analysis Services Training with 7 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers 20 .NET memory dump analysis patterns plus additional unmanaged patterns.
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Publisher: OpenTask (November 2011)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 204 pages
• ISBN-13: 978-1908043320

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the new initiatives for 2012 is the development of SPDL (Software Problem Description Language). Its purpose is automatic generation of a software troubleshooting tool(s) based on the description of a problem. Here software problem means a post-construction problem as outlined in Introduction to Pattern-Driven Software Problem Solving. The tool construction will utilize an expanded set of DebugWare and Workaround patterns together with the refind version of RADII software development process. This will also provide necessary effectiveness, efficiency and enhanced problem solving capabilities to existing TaaS (Tools as a Service) implementations that are limited in the number of tools they offer.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Hidden Parameter pattern is a variant of Execution Residue and String Parameter where we have parameters left out from stack trace due to register calling conventions and compiler optimizations. However, raw stack analysis in a region around stack frames of interest we find what we are looking for. Here’s an example from an x64 system blocked thread waiting for data from a named pipe:

0: kd> kL
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
fffffa60`2c3627d0 fffff800`018b90fa nt!KiSwapContext+0x7f
fffffa60`2c362910 fffff800`018add3b nt!KiSwapThread+0x13a
fffffa60`2c362980 fffff800`01b2121f nt!KeWaitForSingleObject+0x2cb
fffffa60`2c362a10 fffff800`01b319b6 nt!IopXxxControlFile+0xdeb
fffffa60`2c362b40 fffff800`018b68f3 nt!NtFsControlFile+0x56
fffffa60`2c362bb0 00000000`778d6eaa nt!KiSystemServiceCopyEnd+0x13
00000000`11f4da68 00000000`77767b6e ntdll!ZwFsControlFile+0xa
00000000`11f4da70 000007fe`ff94abc8 kernel32!WaitNamedPipeW+0×22f
00000000`11f4db60 000007fe`ff98a32d RPCRT4!NdrProxyForwardingFunction255+0×814d
00000000`11f4dc30 000007fe`ff98918b RPCRT4!OSF_CCONNECTION::TransOpen+0xcd
00000000`11f4dcc0 000007fe`ff988f9b RPCRT4!OSF_CCONNECTION::OpenConnectionAndBind+0×17b
00000000`11f4dd90 000007fe`ff988ec6 RPCRT4!OSF_CCALL::BindToServer+0xbb
00000000`11f4de40 000007fe`ff983368 RPCRT4!OSF_BINDING_HANDLE::InitCCallWithAssociation+0xa5
00000000`11f4dea0 000007fe`ff983220 RPCRT4!OSF_BINDING_HANDLE::AllocateCCall+0×118
00000000`11f4dfd0 000007fe`ffa1f740 RPCRT4!OSF_BINDING_HANDLE::NegotiateTransferSyntax+0×30
00000000`11f4e020 000007fe`ffa1fecb RPCRT4!Ndr64pClientSetupTransferSyntax+0×200
00000000`11f4e080 000007fe`ffa20281 RPCRT4!NdrpClientCall3+0×6b
00000000`11f4e2d0 000007fe`fe087c8c RPCRT4!NdrClientCall3+0xdd
[…]

Even if we disassemble the return address of a caller of WaitNamedPipeW function we won’t easily find the passed first string parameter (named pipe name) unless we do a substancial reverse engineering and data flow analysis:

0: kd> ub 000007fe`ff94abc8
RPCRT4!_imp_load_getaddrinfo+0×7:
000007fe`ff94ab9f jmp     RPCRT4!_tailMerge_WS2_32_dll (000007fe`ff94cef8)
000007fe`ff94aba4 call    qword ptr [RPCRT4!_imp_GetLastError (000007fe`ffa2d528)]
000007fe`ff94abaa mov     r12d,eax
000007fe`ff94abad cmp     r12d,0E7h
000007fe`ff94abb4 jne     RPCRT4!NdrProxyForwardingFunction255+0×8193 (000007fe`ff99c8fb)
000007fe`ff94abba mov     edx,3E8h
000007fe`ff94abbf mov     rcx,rsi
000007fe`ff94abc2 call    qword ptr [RPCRT4!_imp_WaitNamedPipeW (000007fe`ffa2d3f8)]

So dumping raw stack date around corresponding frames give us pipe name clue and possible service to look further:

0: kd> dpu 00000000`11f4da70
00000000`11f4da70  00000000`11f4dba8 “\\.\PIPE\ServiceArpc”
00000000`11f4da78  00000000`00000000
00000000`11f4da80  00000000`00000000
00000000`11f4da88  00000000`000003e8
00000000`11f4da90  00000000`11f4db30
00000000`11f4da98  00000000`00110018
00000000`11f4daa0  00000000`0d9001a0
00000000`11f4daa8  00000000`0000001a
00000000`11f4dab0  00000000`00000000
00000000`11f4dab8  00000000`00000000
00000000`11f4dac0  00000000`0020000c
00000000`11f4dac8  00000000`0d9001e2 “ServiceArpc”
00000000`11f4dad0  00000000`00000000
00000000`11f4dad8  00000000`00000000
00000000`11f4dae0  00000000`00240022

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was recently asked to provide explanation on how to analyze the output of !locks command in kernel and complete memory dumps and I compiled these patterns:

• _ERESOURCE Deadlock
• Mixed Object Deadlock
• Self Deadlock
• High Contention
• Accidental Lock
• _ERESOURCE Wait Chain
• Swarm of Shared Locks
• Semantic Split
• Livelock

and case studies:

http://www.dumpanalysis.org/blog/index.php/2008/07/17/multiple-patterns-in-a-dump-case-study/

http://www.dumpanalysis.org/blog/index.php/2008/10/29/10-common-mistakes-in-memory-analysis-part-3/

http://www.dumpanalysis.org/blog/index.php/2008/12/12/wait-chain-and-spiking-thread-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/02/16/insufficient-memory-handle-leak-process-factory-high-contention-and-busy-system-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/03/03/busy-system-blocked-threads-wait-chains-and-deadlock-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/12/17/wait-chain-blocked-thread-waiting-thread-time-irp-distribution-anomaly-and-stack-trace-collection-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/02/27/inconsistent-dump-stack-trace-collection-lpc-thread-process-executive-resource-wait-chains-missing-threads-and-waiting-thread-time-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/06/03/succession-of-patterns-part-2/

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The slides are available from Memory Dump Analysis Services:

http://www.dumpanalysis.com/Training/Accelerated-NET-Memory-Dump-Analysis-Public.pdf

It also offers the new training sessions in January, 2012:

http://www.dumpanalysis.com/accelerated-net-memory-dump-analysis

http://www.dumpanalysis.com/accelerated-windows-memory-dump-analysis

Also the registration for Advanced training session in December is still open:

http://www.dumpanalysis.com/advanced-windows-memory-dump-analysis

There are also new coming courses for 2012 so stay tuned!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:

• 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
• 4 new pattern interaction case studies
• 11 new trace analysis patterns
• New Debugware pattern
• Introduction to UI problem analysis patterns
• Introduction to intelligence analysis patterns
• Introduction to unified debugging pattern language
• Introduction to generative debugging, metadefect template library and DNA of software behaviour
• The new school of debugging and trends
• .NET memory dump analysis checklist
• Software trace analysis checklist
• Introduction to close and deconstructive readings of a software trace
• Memory dump analysis compass
• Computical and Stack Trace Art
• The abductive reasoning of Philip Marlowe
• Orbifold memory space and cloud computing
• Memory worldview
• Interpretation of cyberspace
• Relationship of memory dumps to religion
• Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5

Product information:

• Title: Memory Dump Analysis Anthology, Volume 6
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 300 pages
• Publisher: Opentask (December 2011)
• ISBN-13: 978-1-908043-19-1

• Hardcover: 300 pages
• Publisher: Opentask (January 2012)
• ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have Linked Messages through some common parameter or attribute. One such example can be found in ETW traces related to kernel process creation notifications. Here we got adjoint thread for module PIDNotify:

#      Module    PID  TID  Time         Message
[...]
128762 PIDNotify 1260 6208 15:53:15.691 Create: ParentID 0x000004EC PID 0×000018D4
[…]
128785 PIDNotify 6356 6388 15:53:15.693 Load: ImageName \Device\HarddiskVolume1\Windows\System32\abscript.exe PID 0×000018D4
[…]
131137 PIDNotify 6356 4568 15:53:15.936 Create: ParentID 0×000018D4 PID 0×00001888
[…]
131239 PIDNotify 6280 6376 15:53:15.958 Load: ImageName \Device\HarddiskVolume1\Windows\System32\wscript.exe PID 0×00001888
[…]
132899 PIDNotify 6356 5704 15:53:16.462 Create: ParentID 0×000018D4 PID 0×00001FD0
[…]
132906 PIDNotify 8144 7900 15:53:16.464 Load: ImageName \Device\HarddiskVolume1\Windows\System32\cmd.exe PID 0×00001FD0
[…]

We see that messages 128762 and 128785 are linked through PID parameter and linked to messages 131137 and 132899 through PID - ParentID parameter relationship. Similar linkages exist for messages 131137 / 131239 and 132899 / 132906.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When analyzing memory dumps from specific application platforms we see threads having definite purpose as part of that specific platform architecture, design and implementation. For example, in applications and services involving .NET CLR we see the following Special Threads:

0:000> !Threads -special
ThreadCount:      9
UnstartedThread:  0
BackgroundThread: 7
PendingThread:    0
DeadThread:       1
Hosted Runtime:   no
                                   PreEmptive   GC Alloc                Lock
       ID  OSID ThreadOBJ    State GC           Context       Domain   Count APT Exception
   0    1   b10 002fbe88      6020 Enabled  0acbdebc:0acbf5a4 002f17d0     0 STA
   2    2   bf0 00306b18      b220 Enabled  00000000:00000000 002f17d0     0 MTA (Finalizer)
   3    3   b34 0034c188      b220 Enabled  00000000:00000000 002f17d0     0 MTA
XXXX    5       0037e3e0     19820 Enabled  00000000:00000000 002f17d0     0 Ukn
   5    7   700 04b606c8   200b220 Enabled  00000000:00000000 002f17d0     0 MTA
   6    4   ec4 04baffa0   200b220 Enabled  00000000:00000000 002f17d0     0 MTA
   8    8   10c 04bf19b8   8009220 Enabled  00000000:00000000 002f17d0     0 MTA (Threadpool Completion Port)
   9   11   464 0be106d8      1220 Enabled  00000000:00000000 002f17d0     0 Ukn
  10   10   da0 003c7958      7220 Disabled 00000000:00000000 0be1dd00     0 STA

       OSID     Special thread type
    1    c08    DbgHelper
    2    bf0    Finalizer
    7    f54    Gate
    8    10c    IOCompletion
    9    464    ADUnloadHelper

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here’s a managed heap counterpart to process heap and kernel pool corruption patterns. It is usually detected by CLR during GC phases. Here is a typical stack from CLR 2 (CLR 4 is similar):

0:000> kL
ChildEBP RetAddr
002baae0 779b06a0 ntdll!KiFastSystemCallRet
002baae4 772c77d4 ntdll!NtWaitForSingleObject+0xc
002bab54 772c7742 kernel32!WaitForSingleObjectEx+0xbe
002bab68 7a0c0a43 kernel32!WaitForSingleObject+0x12
002bab98 7a0c0e89 mscorwks!ClrWaitForSingleObject+0x24
002bb054 7a0c2bfd mscorwks!RunWatson+0x1df
002bb798 7a0c3171 mscorwks!DoFaultReportWorker+0xb62
002bb7d4 7a106b2d mscorwks!DoFaultReport+0xc3
002bb7fc 7a1061ac mscorwks!WatsonLastChance+0x43
002bbcb8 7a10624f mscorwks!EEPolicy::LogFatalError+0x3ae
002bbcd0 79ffee2f mscorwks!EEPolicy::HandleFatalError+0x36
002bbcf4 79f04f1f mscorwks!CLRVectoredExceptionHandlerPhase3+0xc1
002bbd28 79f04e98 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
002bbd5c 79f9149e mscorwks!CLRVectoredExceptionHandler+0x10a
002bbd70 779b1039 mscorwks!CLRVectoredExceptionHandlerShimX86+0x27
002bbd94 779b100b ntdll!ExecuteHandler2+0x26
002bbe3c 779b0e97 ntdll!ExecuteHandler+0x24
002bbe3c 79f69360 ntdll!KiUserExceptionDispatcher+0xf
002bc13c 79f663f1 mscorwks!SVR::heap_segment_next_rw+0xf
002bc228 79f65d63 mscorwks!WKS::gc_heap::plan_phase+0×37c
002bc248 79f6614c mscorwks!WKS::gc_heap::gc1+0×6e
002bc25c 79f65f5d mscorwks!WKS::gc_heap::garbage_collect+0×261
002bc288 79f663c2 mscorwks!WKS::GCHeap::GarbageCollectGeneration+0×1a9
002bc314 79ef1566 mscorwks!WKS::gc_heap::try_allocate_more_space+0×12e
002bc328 79ef1801 mscorwks!WKS::gc_heap::allocate_more_space+0×11
002bc348 79e7510e mscorwks!WKS::GCHeap::Alloc+0×3b
002bc364 79e86713 mscorwks!Alloc+0×60
002bc3a0 79e86753 mscorwks!SlowAllocateString+0×29
002bc3ac 79eb4efb mscorwks!UnframedAllocateString+0xc
002bc3e0 79e91f58 mscorwks!AllocateStringObject+0×2e
002bc424 79e82892 mscorwks!GlobalStringLiteralMap::AddStringLiteral+0×3f
002bc438 79e82810 mscorwks!GlobalStringLiteralMap::GetStringLiteral+0×43
002bc47c 79e82956 mscorwks!AppDomainStringLiteralMap::GetStringLiteral+0×72
002bc494 79e81b6f mscorwks!BaseDomain::GetStringObjRefPtrFromUnicodeString+0×31
002bc4cc 79ef4704 mscorwks!Module::ResolveStringRef+0×88
002bc4e4 79f23132 mscorwks!ConstructStringLiteral+0×39
002bc558 7908c351 mscorwks!CEEInfo::constructStringLiteral+0×108
002bc57c 7906276d mscorjit!Compiler::fgMorphConst+0xa3
002bc598 79065ea0 mscorjit!Compiler::fgMorphTree+0×63
002bc610 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc63c 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc658 79065ea0 mscorjit!Compiler::fgMorphTree+0xa3
002bc6d0 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc6fc 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc718 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bc738 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bc774 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bc788 79064e63 mscorjit!Compiler::fgMorph+0×60
002bc798 790614e6 mscorjit!Compiler::compCompile+0×5f
002bc7f0 79061236 mscorjit!Compiler::compCompile+0×2df
002bc884 7906118c mscorjit!jitNativeCode+0xb8
002bc8bc 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002bc928 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002bc96c 79f0f8da mscorwks!invokeCompileMethod+0×31
002bc9c4 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002bcd7c 79f0e795 mscorwks!UnsafeJitFunction+0×230
002bce20 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002bce78 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002bcec8 00330836 mscorwks!PreStubWorker+0xeb
WARNING: Frame IP not in any known module. Following frames may be wrong.
002bcee0 79e7c74b 0×330836
002bcf10 79e7c6cc mscorwks!CallDescrWorker+0×33
002bcf90 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
002bd0d0 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
002bd0ec 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
002bd100 79e8b983 mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
002bd1d8 79e8b8e6 mscorwks!MethodTable::RunClassInitWorker+0×8b
002bd260 79e8b7fa mscorwks!MethodTable::RunClassInitEx+0×11e
002bd724 79ebcee6 mscorwks!MethodTable::DoRunClassInitThrowing+0×2f0
002bd79c 79fc49db mscorwks!MethodTable::CheckRunClassInitNT+0×8c
002bd82c 790a2801 mscorwks!CEEInfo::initClass+0×19b
002bddcc 79062cdc mscorjit!Compiler::impExpandInline+0×2aaa
002bde24 79062b7c mscorjit!Compiler::fgMorphCallInline+0xf8
002bde50 7906311f mscorjit!Compiler::fgMorphCall+0×27b
002bde6c 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bde8c 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bdec8 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bdedc 79064e63 mscorjit!Compiler::fgMorph+0×60
002bdeec 790614e6 mscorjit!Compiler::compCompile+0×5f
002bdf44 79061236 mscorjit!Compiler::compCompile+0×2df
002bdfd8 7906118c mscorjit!jitNativeCode+0xb8
002be010 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002be07c 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002be0c0 79f0f8da mscorwks!invokeCompileMethod+0×31
002be118 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002be4d0 79f0e795 mscorwks!UnsafeJitFunction+0×230
002be574 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002be5cc 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002be61c 00330836 mscorwks!PreStubWorker+0xeb
002be634 0570c859 0×330836
002be69c 0595bcc1 0×570c859
002be700 0595b954 0×595bcc1
002be704 099b66e0 0×595b954
002be708 002be728 0×99b66e0
002be70c 09589c90 0×2be728
002be728 099b67b8 0×9589c90
002be72c 00000000 0×99b67b8

Usually !VerifyHeap SOS command helps to find the first invalid object on managed heap and shows the last valid one. Sometimes the corruption can deeply affect heap or when a crash happens during traversal GC state might not be valid for analysis:

0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state

0:000> !DumpHeap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state

In such cases it is recommended to collect several dumps to catch more consistent heap state:

0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
object 0981f024: does not have valid MT
curr_object:      0981f024
Last good object: 0981f010
—————-

Then we can use !DumpObj (!do) command to check objects and d* WinDbg commands to inspect raw memory.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -