Archive for the ‘Complete Memory Dump Analysis’ Category

Crash Dump Analysis Patterns (Part 229)

Sunday, September 13th, 2015

The advent of virtual machines, the possibility of saving complete memory snapshots without interruption, and the ability to quickly convert such snapshots into a debugger readable memory dump format such as in the case of VMware allows to study how Stack Trace Collections and Wait Chains change over time in complex problem scenarios. Such Stack Trace Surface may also show service restarts if PID changes for processes of interest. We call this pattern by analogy with a memory dump surface where each line corresponds to an individual memory snapshot with coordinates from 0 to the highest address:

In case of orbifold memory space we have a case of a 3D volume (we may call 3D orbifold).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 27e)

Monday, August 24th, 2015

This is another variant of Stack Trace Collection pattern that shows stack traces from threads currently execution on all CPUs. Although we can see the non-idle running threads from the stack traces corresponding to all processes and their threads we may also want to see idle thread stack traces too. Also, the corresponding WinDbg command (!running -t -i) is faster if we want to double check the output of !analyze -v command in case of BSOD. The latter command may show the stack trace from the current CPU instead of the stack trace from the thread running on a different CPU that caused a bugcheck. Here’s an example from one of the memory dumps for which !analyze -v command shows an incorrect stack trace in the output when we open the dump file. It reports the stack trace from CPU 0 but the bugcheck  happened on CPU 1:

0: kd> !running -t -i

System Processors:  (00000000000000ff)
Idle Processors:  (00000000000000fd)

Prcbs             Current         (pri) Next            (pri) Idle
0    fffff801e5d85180  fffff801e5ddea00 ( 0)                       fffff801e5ddea00  ................

Child-SP          RetAddr           Call Site
fffff801`e8c9eb60 fffff801`e5c69b74 hal!KeQueryPerformanceCounter+0x75
fffff801`e8c9eba0 fffff801`e5c69e01 nt!KiCheckStall+0x2c
fffff801`e8c9ebd0 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
fffff801`e8c9ece0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
fffff801`e8c9ed30 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
fffff801`e8c9ee70 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
fffff801`e8c8c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
fffff801`e8c8c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
fffff801`e8c8c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
fffff801`e8c8cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
fffff801`e8c8cc60 00000000`00000000 nt!KiIdleLoop+0x2c

1    ffffd000f0975180  ffffe0000d726880 (12)                       ffffd000f09813c0  …………….

Child-SP          RetAddr           Call Site
ffffd000`202cb618 fffff801`e5a1cc3c hal!HalpAcpiPmRegisterReadPort+0x1b
ffffd000`202cb620 fffff801`e5a417e7 hal!HalpAcpiPmRegisterRead+0x30
ffffd000`202cb650 fffff801`e5c66af5 hal!HaliHaltSystem+0x53
ffffd000`202cb690 fffff801`e5c66741 nt!KiBugCheckDebugBreak+0×99
ffffd000`202cb6f0 fffff801`e5bd2aa4 nt!KeBugCheck2+0xc6d
ffffd000`202cbe00 fffff801`e5bde4e9 nt!KeBugCheckEx+0×104
ffffd000`202cbe40 fffff801`e5bdcd3a nt!KiBugCheckDispatch+0×69
ffffd000`202cbf80 fffff800`913601da nt!KiPageFault+0×23a
ffffd000`202cc118 fffff800`91363710 DriverA!memcpy+0×21a

[…]

2    ffffd000f09ee180  ffffd000f09fa3c0 ( 0)                       ffffd000f09fa3c0  ................

Child-SP          RetAddr           Call Site
ffffd000`f09f9f88 fffff801`e5c69e01 nt!KiCheckStall+0xa
ffffd000`f09f9f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`f09fa0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`f09fa0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`f09fa230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb5938e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb5938f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb593920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb593b10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb593c60 00000000`00000000 nt!KiIdleLoop+0x2c

3    ffffd000eb5e5180  ffffd000eb5f13c0 ( 0)                       ffffd000eb5f13c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb5f0f60 fffff801`e5c69e01 nt!KiCheckStall+0x5f
ffffd000`eb5f0f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb5f10a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb5f10f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb5f1230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb5fa8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb5fa8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb5fa920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb5fab10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb5fac60 00000000`00000000 nt!KiIdleLoop+0x2c

4    ffffd000f08d1180  ffffd000f08dd3c0 ( 0)                       ffffd000f08dd3c0  ................

Child-SP          RetAddr           Call Site
ffffd000`f08dcf90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x227
ffffd000`f08dd0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`f08dd0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`f08dd230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb85b8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb85b8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb85b920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb85bb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb85bc60 00000000`00000000 nt!KiIdleLoop+0x2c

5    ffffd000eb8ad180  ffffd000eb8b93c0 ( 0)                       ffffd000eb8b93c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb8b8f60 fffff801`e5c69e01 nt!KiCheckStall+0x75
ffffd000`eb8b8f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb8b90a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb8b90f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb8b9230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb8db8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb8db8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb8db920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb8dbb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb8dbc60 00000000`00000000 nt!KiIdleLoop+0x2c

6    ffffd000eb92a180  ffffd000eb9363c0 ( 0)                       ffffd000eb9363c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb935f60 fffff801`e5c69e01 nt!KiCheckStall+0x75
ffffd000`eb935f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb9360a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb9360f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb936230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb93f8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb93f8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb93f920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb93fb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb93fc60 00000000`00000000 nt!KiIdleLoop+0x2c

7    ffffd000eb967180  ffffd000eb9733c0 ( 0)                       ffffd000eb9733c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb972f60 fffff801`e5c69e01 nt!KiCheckStall+0x75
ffffd000`eb972f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb9730a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb9730f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb973230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb97c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb97c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb97c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb97cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb97cc60 00000000`00000000 nt!KiIdleLoop+0x2c

This command is obviously faster than repeatedly switching to subsequent CPUs using ~s command and then checking the corresponding stack trace (k). It also helps in diagnosing Spiking Threads in kernel and complete memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 218)

Wednesday, December 31st, 2014

Objects such as processes may be referenced internally in addition to using handles. If their reference counts are unbalanced we may have Reference Leak pattern. For example, we have an instance of thousands of Zombie Processes but we don’t see Handle Leaks from their parent processes if we analyze ParentCids:

0: kd> !process 0 0
[...]
PROCESS fffffa801009a060
SessionId: 0 Cid: 2e270 Peb: 7fffffdb000 ParentCid: 032c
DirBase: 12ba37000 ObjectTable: 00000000 HandleCount: 0.
Image: conhost.exe

PROCESS fffffa8009b7e8e0
SessionId: 1 Cid: 2e0c8 Peb: 7fffffd9000 ParentCid: 10a0
DirBase: 21653e000 ObjectTable: 00000000 HandleCount: 0.
Image: taskmgr.exe

PROCESS fffffa8009e7a450
SessionId: 0 Cid: 2e088 Peb: 7efdf000 ParentCid: 0478
DirBase: 107f02000 ObjectTable: 00000000 HandleCount: 0.
Image: AppA.exe

PROCESS fffffa8009e794b0
SessionId: 0 Cid: 2e394 Peb: 7fffffd3000 ParentCid: 032c
DirBase: 210ffc000 ObjectTable: 00000000 HandleCount: 0.
Image: conhost.exe

PROCESS fffffa8009ed4060
SessionId: 0 Cid: 2dee4 Peb: 7efdf000 ParentCid: 0478
DirBase: 11b7c7000 ObjectTable: 00000000 HandleCount: 0.
Image: AppB.exe

PROCESS fffffa800a13bb30
SessionId: 0 Cid: 2e068 Peb: 7fffffd5000 ParentCid: 032c
DirBase: 1bb8c1000 ObjectTable: 00000000 HandleCount: 0.
Image: conhost.exe

PROCESS fffffa80096f26b0
SessionId: 0 Cid: 2e320 Peb: 7efdf000 ParentCid: 0478
DirBase: 6ad4c000 ObjectTable: 00000000 HandleCount: 0.
Image: AppC.exe

PROCESS fffffa8009c44060
SessionId: 0 Cid: 2e300 Peb: 7fffffdd000 ParentCid: 032c
DirBase: 10df06000 ObjectTable: 00000000 HandleCount: 0.
Image: conhost.exe
[...]

0: kd> !object fffffa800a13bb30
Object: fffffa800a13bb30 Type: (fffffa8006cecf30) Process
ObjectHeader: fffffa800a13bb00 (new version)
HandleCount: 0 PointerCount: 1

0: kd> !object fffffa8009b7e8e0
Object: fffffa8009b7e8e0 Type: (fffffa8006cecf30) Process
ObjectHeader: fffffa8009b7e8b0 (new version)
HandleCount: 0 PointerCount: 1

Such number of processes correlates with non-paged pool usage for process structures:

0: kd> !poolused 3
....
Sorting by NonPaged Pool Consumed

NonPaged Paged
Tag Allocs Frees Diff Used Allocs Frees Diff Used

Proc 55488 60 55428 80328320 0 0 0 0 Process objects , Binary: nt!ps
File 51733526 51708737 24789 7150416 0 0 0 0 File objects
[…]

Here we recommend enabling object reference tracing either using gflags.exe or directly modifying registry:

Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel
Value: ObTracePoolTags
Type: REG_SZ
Data: Proc

Note: after troubleshooting or debugging please disable tracing because it consumes pool (another variant of Instrumentation Side Effect pattern and may lead to similar Insufficient Memory pattern for stack trace database):

0: kd> !poolused 3
....
Sorting by NonPaged Pool Consumed

NonPaged Paged
Tag Allocs Frees Diff Used Allocs Frees Diff Used

ObRt 5688634 5676109 12525 4817288240 0 0 0 0 object reference stack tracing , Binary: nt!ob
Proc 22120 101 22019 25961168 0 0 0 0 Process objects , Binary: nt!ps
[…]

After enabling tracing we collect a complete memory dump (in case of postmortem debugging) to analyze another variant of Stack Trace pattern using !obtrace WinDbg command:

0: kd> !obtrace fffffa800af9e220
Object: fffffa800af9e220
Image: AppD.exe
Sequence (+/-) Tag Stack
-------- ----- ---- ---------------------------------------------------
ad377858 +1 Dflt nt! ?? ::NNGAKEGL::`string'+21577
nt!PspAllocateProcess+185
nt!NtCreateUserProcess+4a3
nt!KiSystemServiceCopyEnd+13

ad37787d +1 Dflt nt! ?? ::FNODOBFM::`string'+18f1d
nt!NtCreateUserProcess+569
nt!KiSystemServiceCopyEnd+13

ad377882 +1 Dflt nt! ?? ::NNGAKEGL::`string'+1f9d8
nt!NtProtectVirtualMemory+119
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
nt!RtlCreateUserStack+1e4
nt!PspAllocateThread+299
nt!NtCreateUserProcess+65d
nt!KiSystemServiceCopyEnd+13

ad377884 -1 Dflt nt! ?? ::FNODOBFM::`string'+4886e
nt!NtProtectVirtualMemory+161
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
nt!RtlCreateUserStack+1e4
[...]

Analysis of such traces may be complicated due to Truncated Stack Traces. We plan to show one counting trick in the next pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 215)

Sunday, November 16th, 2014

Sometimes Wait Chains such as involving critical sections may have a Missing Thread endpoint. But in some cases we might see a Ghost Thread whose TID was reused by subsequent thread creation in a different process. For example, critical section structure may refer to such TID as in the example below.

// Critical section from LSASS process

THREAD fffffa803431cb50 Cid 03e8.2718 Teb: 000007fffff80000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa80330e0500 SynchronizationEvent
Impersonation token: fffff8a00b807060 (Level Impersonation)
Owning Process            fffffa8032354c40     Image: lsass.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      107175         Ticks: 19677 (0:00:05:06.963)
Context Switch Count      2303           IdealProcessor: 1
UserTime                  00:00:00.218
KernelTime                00:00:00.109
Win32 Start Address ntdll!TppWorkerThread (0×0000000076e1f2e0)
Stack Init fffff88008e5fdb0 Current fffff88008e5f900
Base fffff88008e60000 Limit fffff88008e5a000 Call 0
Priority 10 BasePriority 10 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP RetAddr Call Site
fffff880`08e5f940 fffff800`01c7cf72 nt!KiSwapContext+0×7a
fffff880`08e5fa80 fffff800`01c8e39f nt!KiCommitThreadWait+0×1d2
fffff880`08e5fb10 fffff800`01f7fe3e nt!KeWaitForSingleObject+0×19f
fffff880`08e5fbb0 fffff800`01c867d3 nt!NtWaitForSingleObject+0xde
fffff880`08e5fc20 00000000`76e5067a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`08e5fc20)
00000000`0427cca8 00000000`76e4d808 ntdll!NtWaitForSingleObject+0xa
00000000`0427ccb0 00000000`76e4d6fb ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`0427cd60 000007fe`f46a4afe ntdll!RtlEnterCriticalSection+0xd1

[…]

1: kd> .process /r /p fffffa8032354c40
Implicit process is now fffffa80`32353b30
Loading User Symbols

1: kd> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x0000000003475220
Critical section   = 0x0000000003377740 (+0x3377740)
LOCKED
LockCount          = 0×10
WaiterWoken        = No
OwningThread       = 0×00000000000004e4
RecursionCount     = 0×0
LockSemaphore      = 0×0
SpinCount          = 0×0000000000000000
OwningThread       = .thread fffffa80344e4c00
[…]

// The "owner" thread is from winlogon.exe

1: kd> !thread fffffa80344e4c00 3f
THREAD fffffa80344e4c00 Cid 21d0.14e4 Teb: 000007fffffae000 Win32Thread: fffff900c0998c20 WAIT: (WrUserRequest) UserMode Non-Alertable
fffffa80355817d0 SynchronizationEvent
Not impersonating
DeviceMap            fffff8a0000088f0
Owning Process       fffffa8034ff77c0       Image: winlogon.exe
[…]

A PML (Process Monitor) log was recorded before the complete memory dump was forced, and it clearly shows Glued Activity trace analysis pattern. The thread was owned by LSASS but then exited and its TID was subsequently reused by 2 other processes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 204)

Friday, April 18th, 2014

One of useful patterns for the analysis of system hangs is Thread Waiting Time. If there are many such threads of interest they can be partitioned by waiting time and modules of interest from their stack traces. Modules of interest may include Directing, Coupled, Blocking, Top, and/or Problem modules depending on the problem description. We call the resulting composite pattern Thread Cluster. Extra-dimensional information can also be added such as the number of threads having the same or similar waiting time and other attributes by using different colours. For example, on this diagram, illustrating a real system hang, we see clustering of threads running through one 3rd-party module of interest and having the longest waiting time. Also we are able to identify possibly coupled (semantically related) threads running through another module of interest:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 42l)

Wednesday, April 9th, 2014

This is a variation of a general Wait Chain pattern related to CLR threads. When looking at Stack Trace Collection from a complete memory dump we may find threads using a monitor synchronization mechanism:

[... 32-bit ...]
09d2e908 6ba4d409 clr!CLREvent::WaitEx+0x106
09d2e91c 6bb90160 clr!CLREvent::Wait+0x19
09d2e9ac 6bb90256 clr!AwareLock::EnterEpilogHelper+0xa8
09d2e9ec 6bb9029b clr!AwareLock::EnterEpilog+0x42
09d2ea0c 6ba90f78 clr!AwareLock::Enter+0x5f
09d2eaa8 05952499 clr!JIT_MonEnterWorker_Portable+0xf8
[…]

or

[... 64-bit ...]
00000000`2094e230 000007fe`eedc3e3a clr!CLREvent::WaitEx+0xc1
00000000`2094e2d0 000007fe`eedc3d43 clr!AwareLock::EnterEpilogHelper+0xca
00000000`2094e3a0 000007fe`eee3e613 clr!AwareLock::EnterEpilog+0x63
00000000`2094e400 000007ff`007f4c38 clr!JIT_MonEnterWorker_Portable+0×14f
[…]

When seeing such threads we may ask for a process memory dump to perform .NET memory dump analysis using SOS or other WinDbg extensions such as in Deadlock pattern example for CLR 2 (mscorwks).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 200)

Friday, August 16th, 2013

If you have found module related patterns in a complete memory dump and suspect a particular module it may be worth looking at Module Product Process if it exists especially if this module (component, DLL) has product information or some related hint (lmv or !lmi commands). In complex environments such modules may be loaded not only by hooking mechanisms but also as plugins. If you are not sure if there is any such process the best ways is to get module collection and find a process module that has the same vendor as the module in question. Then such process should also be analysed for anomalies.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 198)

Sunday, May 26th, 2013

All previous wait chain patterns were about single wait chains. However, it is often a case when there are many different wait chains in a memory dump especially in terminal services environments. There can be ALPC and critical section wait chains at the same time. The can be related or completely disjoint. Here we call a special case of several wait chains having the same structure (and possibly pointing to one direction) Distributed Wait Chain. One such example we put below. In a stack trace collection from a complete memory dump from a hanging system we found several explorer.exe processes with critical section wait chains having the same structure and endpoint of a top and blocking ModuleA:

THREAD fffffa80137cf060  Cid 4884.4f9c  Teb: 000007fffffaa000 Win32Thread: fffff900c0fb98b0 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8013570dc0  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a014e21d90
Owning Process            fffffa80131a75d0       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      274752         Ticks: 212448 (0:00:55:19.500)
Context Switch Count      9889                 LargeStack
UserTime                  00:00:00.093
KernelTime                00:00:00.171
Win32 Start Address SHLWAPI!WrapperThreadProc (0×000007fefdafc608)
Stack Init fffff88013c25db0 Current fffff88013c25900
Base fffff88013c26000 Limit fffff88013c1b000 Call 0
Priority 11 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`13c25940 fffff800`01873652 nt!KiSwapContext+0×7a
fffff880`13c25a80 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2
fffff880`13c25b10 fffff800`01b7768e nt!KeWaitForSingleObject+0×19f
fffff880`13c25bb0 fffff800`0187ced3 nt!NtWaitForSingleObject+0xde
fffff880`13c25c20 00000000`76d8135a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`13c25c20)
00000000`0489e518 00000000`76d7e4e8 ntdll!ZwWaitForSingleObject+0xa
00000000`0489e520 00000000`76d7e3db ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`0489e5d0 000007fe`fdf8ff50 ntdll!RtlEnterCriticalSection+0xd1
00000000`0489e600 000007fe`fdf8fbd3 SHELL32!CFSFolder::GetIconOf+0×24b
00000000`0489f3a0 000007fe`fdf903d3 SHELL32!SHGetIconIndexFromPIDL+0×3f
00000000`0489f3d0 00000000`ff900328 SHELL32!SHMapIDListToSystemImageListIndexAsync+0×73
00000000`0489f470 00000000`ff8fff4b Explorer!SFTBarHost::AddImageForItem+0×9c
00000000`0489f4d0 00000000`ff8fd2f1 Explorer!SFTBarHost::_InternalRepopulateList+0×4ad
00000000`0489f5d0 00000000`ff8fd0b4 Explorer!SFTBarHost::_RepopulateList+0×1f3
00000000`0489f600 00000000`ff8fcccd Explorer!SFTBarHost::_OnBackgroundEnumDone+0xc1
00000000`0489f630 00000000`ff8fc9e2 Explorer!SFTBarHost::_WndProc+0×451
00000000`0489f680 00000000`76669bd1 Explorer!SFTBarHost::_WndProc_ProgramsMFU+0×1b
00000000`0489f6b0 00000000`766698da USER32!UserCallWinProcCheckWow+0×1ad
00000000`0489f770 00000000`ff8f1177 USER32!DispatchMessageWorker+0×3b5
00000000`0489f7f0 00000000`ff9130e9 Explorer!CTray::_MessageLoop+0×446
00000000`0489f880 000007fe`fdafc71e Explorer!CTray::MainThreadProc+0×8a
00000000`0489f8b0 00000000`76c2652d SHLWAPI!WrapperThreadProc+0×19b
00000000`0489f9b0 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd
00000000`0489f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .process /r /p fffffa80131a75d0
Implicit process is now fffffa80`131a75d0
Loading User Symbols

0: kd> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x0000000000499d90
Critical section   = 0x000007fefe3d5900 (SHELL32!g_csIconCache+0x0)
LOCKED
LockCount          = 0×2
WaiterWoken        = No
OwningThread       = 0×0000000000002b34
RecursionCount     = 0×1
LockSemaphore      = 0×7F8
SpinCount          = 0×0000000000000000
OwningThread       = .thread fffffa8013dc3b00

THREAD fffffa8013dc3b00  Cid 4884.2b34  Teb: 000007fffffac000 Win32Thread: fffff900c2bc1010 WAIT: (Executive) KernelMode Non-Alertable
    fffff88011c03600  SynchronizationEvent
IRP List:
    fffffa800f8fc790: (0006,0430) Flags: 00000404  Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a014e21d90
Owning Process            fffffa80131a75d0       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      170052         Ticks: 317148 (0:01:22:35.437)
Context Switch Count      2                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address SHELL32!ShutdownThreadProc (0x000007fefe13ef54)
Stack Init fffff88011c03db0 Current fffff88011c03320
Base fffff88011c04000 Limit fffff88011bfd000 Call 0
Priority 11 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`11c03360 fffff800`01873652 nt!KiSwapContext+0x7a
fffff880`11c034a0 fffff800`01884a9f nt!KiCommitThreadWait+0x1d2
fffff880`11c03530 fffff880`05c12383 nt!KeWaitForSingleObject+0x19f
fffff880`11c035d0 fffff880`012b9288 ModuleA+0×12468
fffff880`11c03750 fffff880`012b7d1b fltmgr!FltpPerformPostCallbacks+0×368
fffff880`11c03820 fffff880`012b66df fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×39b
fffff880`11c038b0 fffff880`01b895ff fltmgr!FltpDispatch+0xcf
fffff880`11c03a30 fffff800`01b783b4 nt!IopCloseFile+0×11f
fffff880`11c03ac0 fffff800`01b78171 nt!ObpDecrementHandleCount+0xb4
fffff880`11c03b40 fffff800`01b78734 nt!ObpCloseHandleTableEntry+0xb1
fffff880`11c03bd0 fffff800`0187ced3 nt!ObpCloseHandle+0×94
fffff880`11c03c20 00000000`76d8140a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`11c03c20)
00000000`0754f348 000007fe`fd341873 ntdll!NtClose+0xa
00000000`0754f350 00000000`76c32f51 KERNELBASE!CloseHandle+0×13
00000000`0754f380 000007fe`fdaf9690 kernel32!CloseHandleImplementation+0×3d
00000000`0754f490 000007fe`fe191d7f SHLWAPI!CFileStream::Release+0×84
00000000`0754f4c0 000007fe`fe13ed57 SHELL32!IconCacheSave+0×2b7
00000000`0754f780 000007fe`fe13f0c6 SHELL32!CommonRestart+0×2f
00000000`0754f7f0 00000000`76c2652d SHELL32!ShutdownThreadProc+0×172
00000000`0754f820 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd
00000000`0754f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Malware Analysis Patterns (Part 10)

Sunday, January 20th, 2013

We skip parts 5 (Module Collection), 6 (No Component Symbols, for looking at import tables), 7 (Stack Trace Collection, for listing active processes, threads and their stack traces), 8 (Hidden Module), and 9 (Hidden Process). The new pattern here is called Driver Device Collection and can be used to compare the current list of device and driver objects with some saved reference list to find out any changes. This listing can be done by using !object command:

0: kd> !object \Driver
[...]

0: kd> !object \FileSystem
[...]

0: kd> !object \Device
[...]

Note that the collection is called Driver Device and not Device Driver.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 190)

Monday, December 24th, 2012

In addition to stack trace collection we often are interested in Module Collection (we called this pattern initially Vendor Collection), especially if we would like to check if some vendor DLL is present in some process address space in a complete memory dump (kernel module list or module list from a process memory dump is trivial). Or we need to check for some vendor information from problem description (lmv command). If we have a complete memory dump from x64 system then listing modules for each process is not enough. For example, we might have this:

0: kd> lmu
start             end                 module name
00000000`00ab0000 00000000`00ae8000   AppA    (deferred)
00000000`74fe0000 00000000`7502e000   wow64win   (deferred)
00000000`75030000 00000000`75075000   wow64      (deferred)
00000000`750c0000 00000000`750c9000   wow64cpu   (deferred)
00000000`77b70000 00000000`77cf7000   ntdll      (pdb symbols)

AppA is a 32-bit process and has an additional 32-bit module list that is more useful. We can set x86 context for a thread from that process and get the list of 32-bit modules:

0: kd> .load wow64exts

0: kd> .thread /w fffffa800e372060
Implicit thread is now fffffa80`0e372060
x86 context set

0: kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols

0: kd:x86> lmu
start             end                 module name
00000000`00ab0000 00000000`00ae8000   AppA    (deferred)
00000000`73490000 00000000`73515000   COMCTL32   (deferred)
00000000`73520000 00000000`735c3000   MSVCR90    (deferred)
00000000`735d0000 00000000`7365e000   MSVCP90    (deferred)
00000000`74920000 00000000`7493e000   USERENV    (deferred)
00000000`74940000 00000000`74ade000   comctl32_74940000   (deferred)
00000000`74af0000 00000000`74b02000   MSASN1     (deferred)
00000000`74b10000 00000000`74c03000   CRYPT32    (deferred)
00000000`74dc0000 00000000`74e5b000   MSVCR80    (deferred)
00000000`74f60000 00000000`74fd6000   NETAPI32   (deferred)
00000000`74fe0000 00000000`7502e000   wow64win   (deferred)
00000000`75030000 00000000`75075000   wow64      (deferred)
00000000`750b0000 00000000`750ba000   WTSAPI32   (deferred)
00000000`750c0000 00000000`750c9000   wow64cpu   (deferred)
00000000`75cf0000 00000000`75d50000   Secur32    (deferred)
00000000`75d50000 00000000`76861000   SHELL32    (deferred)
00000000`76a10000 00000000`76aa0000   GDI32      (deferred)
00000000`76b30000 00000000`76b90000   IMM32      (deferred)
00000000`76be0000 00000000`76cf0000   kernel32   (deferred)
00000000`76e30000 00000000`76f75000   ole32      (deferred)
00000000`76f80000 00000000`7702a000   msvcrt     (deferred)
00000000`77030000 00000000`77037000   PSAPI      (deferred)
00000000`77040000 00000000`77110000   USER32     (deferred)
00000000`77110000 00000000`77169000   SHLWAPI    (deferred)
00000000`77170000 00000000`771ed000   USP10      (deferred)
00000000`77380000 00000000`7740d000   OLEAUT32   (deferred)
00000000`77640000 00000000`77649000   LPK        (deferred)
00000000`776e0000 00000000`777d0000   RPCRT4     (deferred)
00000000`777d0000 00000000`77898000   MSCTF      (deferred)
00000000`778a0000 00000000`77966000   ADVAPI32   (deferred)
00000000`77b70000 00000000`77cf7000   ntdll      (pdb symbols)
00000000`77d30000 00000000`77e90000   ntdll_77d30000 # (pdb symbols)

So it looks like we need to dump modules for each thread. However, the output would be enormous unless we skip threads having the same PID. After some tinkering I wrote this WinDbg script with moderate output volume:

.load wow64exts
!for_each_thread ".thread @#Thread; .if (@$t0 != @@c++(@$thread->Cid.UniqueProcess)) {.reload /user;lmvu;.thread /w @#Thread;.reload /user;lmvu;r $t0 = @@c++(@$thread->Cid.UniqueProcess);.effmach AMD64; }"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg Reference Cards Version 2 (Page 1)

Thursday, November 15th, 2012

Finally, the new version of WinDbg: A Reference Poster and Learning Cards is under development. This time every page is published online for comments, suggestions and corrections which are very welcome. The format of every page follows colored memory space diagram where red cards are for native kernel space commands, blue cards are for unmanaged user space, and green cards are for managed .NET space (click on a picture to open a PDF file):

Download page 1 PDF file

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 184)

Wednesday, October 31st, 2012

Looks like Windows 8 reuses the debugging concept of a frozen thread for the so called a “deeply frozen” process:

0: kd> !sprocess 2
Dumping Session 2
[...]
PROCESS fffffa8002cb2940
SessionId: 2  Cid: 0c80    Peb: 7f6c41dd000  ParentCid: 0288
DeepFreeze
DirBase: 2ef45000  ObjectTable: fffff8a002f215c0  HandleCount: <Data Not Accessible>
Image: iexplore.exe
[…]

0: kd> dt nt!_KPROCESS fffffa8002cb2940
+0x000 Header           : _DISPATCHER_HEADER
+0x018 ProfileListHead  : _LIST_ENTRY [ 0xfffffa80`02cb2958 - 0xfffffa80`02cb2958 ]
+0x028 DirectoryTableBase : 0x2ef45000
+0x030 ThreadListHead   : _LIST_ENTRY [ 0xfffffa80`01e4edf8 - 0xfffffa80`01f5bbf8 ]
+0x040 ProcessLock      : 0
+0x044 Spare0           : 0
+0x048 Affinity         : _KAFFINITY_EX
+0x0f0 ReadyListHead    : _LIST_ENTRY [ 0xfffffa80`02cb2a30 - 0xfffffa80`02cb2a30 ]
+0x100 SwapListEntry    : _SINGLE_LIST_ENTRY
+0x108 ActiveProcessors : _KAFFINITY_EX
+0x1b0 AutoAlignment    : 0y0
+0x1b0 DisableBoost     : 0y0
+0x1b0 DisableQuantum   : 0y0
+0x1b0 AffinitySet      : 0y0
+0×1b0 DeepFreeze       : 0y1
+0×1b0 TimerVirtualization : 0y1
+0×1b0 ActiveGroupsMask : 0y00000000000000000001 (0×1)
+0×1b0 ReservedFlags    : 0y000000 (0)
+0×1b0 ProcessFlags     : 0n112
+0×1b4 BasePriority     : 8 ”
+0×1b5 QuantumReset     : 6 ”
+0×1b6 Visited          : 0 ”
+0×1b7 Flags            : _KEXECUTE_OPTIONS
+0×1b8 ThreadSeed       : [20] 0
+0×208 IdealNode        : [20] 0
+0×230 IdealGlobalNode  : 0
+0×232 Spare1           : 0
+0×234 StackCount       : _KSTACK_COUNT
+0×238 ProcessListEntry : _LIST_ENTRY [ 0xfffffa80`03816b78 - 0xfffffa80`02cc2b78 ]
+0×248 CycleTime        : 0×225078
+0×250 ContextSwitches  : 0×22
+0×258 SchedulingGroup  : (null)
+0×260 FreezeCount      : 0
+0×264 KernelTime       : 0
+0×268 UserTime         : 0
+0×26c LdtFreeSelectorHint : 0
+0×26e LdtTableLength   : 0
+0×270 LdtSystemDescriptor : _KGDTENTRY64
+0×280 LdtBaseAddress   : (null)
+0×288 LdtProcessLock   : _FAST_MUTEX
+0×2c0 InstrumentationCallback : (null)

We also see that all its threads have a freeze count 1:

0: kd> !process fffffa8002cb2940 2
[...]
THREAD fffffa8001e4eb00  Cid 0c80.0514  Teb: 000007f6c41de000 Win32Thread: fffff901000e5b90 WAIT: (Suspended) KernelMode Non-Alertable
FreezeCount 1
fffffa8001e4ede0  NotificationEvent

THREAD fffffa800219c080  Cid 0c80.0d88  Teb: 000007f6c41db000 Win32Thread: fffff90103f206e0 WAIT: (Suspended) KernelMode Non-Alertable
FreezeCount 1
fffffa800219c360  NotificationEvent
[…]

0: kd> dt _KTHREAD fffffa800219c080
nt!_KTHREAD
+0x000 Header           : _DISPATCHER_HEADER
+0x018 SListFaultAddress : (null)
+0x020 QuantumTarget    : 0x18c26200
+0x028 InitialStack     : 0xfffff880`1548ddd0 Void
+0x030 StackLimit       : 0xfffff880`15488000 Void
+0x038 StackBase        : 0xfffff880`1548e000 Void
+0x040 ThreadLock       : 0
+0x048 CycleTime        : 0x15ca97c8
+0x050 CurrentRunTime   : 0
+0x054 ExpectedRunTime  : 0xd77e
+0x058 KernelStack      : 0xfffff880`1548d430 Void
+0x060 StateSaveArea    : 0xfffff880`1548de00 _XSAVE_FORMAT
+0x068 SchedulingGroup  : (null)
+0x070 WaitRegister     : _KWAIT_STATUS_REGISTER
+0x071 Running          : 0 ''
+0x072 Alerted          : [2]  ""
+0x074 KernelStackResident : 0y1
+0x074 ReadyTransition  : 0y0
+0x074 ProcessReadyQueue : 0y0
+0x074 WaitNext         : 0y0
+0x074 SystemAffinityActive : 0y0
+0x074 Alertable        : 0y0
+0x074 CodePatchInProgress : 0y0
+0x074 UserStackWalkActive : 0y0
+0x074 ApcInterruptRequest : 0y0
+0x074 QuantumEndMigrate : 0y0
+0x074 UmsDirectedSwitchEnable : 0y0
+0x074 TimerActive      : 0y0
+0x074 SystemThread     : 0y0
+0x074 ProcessDetachActive : 0y0
+0x074 CalloutActive    : 0y0
+0x074 ScbReadyQueue    : 0y0
+0x074 ApcQueueable     : 0y1
+0x074 ReservedStackInUse : 0y0
+0x074 UmsPerformingSyscall : 0y0
+0x074 Reserved         : 0y0000000000000 (0)
+0x074 MiscFlags        : 0n65537
+0x078 AutoAlignment    : 0y0
+0x078 DisableBoost     : 0y0
+0x078 UserAffinitySet  : 0y0
+0x078 AlertedByThreadId : 0y0
+0x078 QuantumDonation  : 0y0
+0x078 EnableStackSwap  : 0y1
+0x078 GuiThread        : 0y1
+0x078 DisableQuantum   : 0y0
+0x078 ChargeOnlyGroup  : 0y0
+0x078 DeferPreemption  : 0y0
+0x078 QueueDeferPreemption : 0y0
+0x078 ForceDeferSchedule : 0y0
+0x078 ExplicitIdealProcessor : 0y0
+0×078 FreezeCount      : 0y1
+0×078 EtwStackTraceApcInserted : 0y00000000 (0)
+0×078 ReservedFlags    : 0y0000000000 (0)
+0×078 ThreadFlags      : 0n8288
+0×07c Spare0           : 0
+0×080 SystemCallNumber : 0×87
+0×084 Spare1           : 0
+0×088 FirstArgument    : 0×00000000`0000017c Void
+0×090 TrapFrame        : (null)
+0×098 ApcState         : _KAPC_STATE
+0×098 ApcStateFill     : [43]  “???”
+0×0c3 Priority         : 8 ”
+0×0c4 UserIdealProcessor : 1
+0×0c8 WaitStatus       : 0n256
+0×0d0 WaitBlockList    : 0xfffffa80`0219c1c0 _KWAIT_BLOCK
+0×0d8 WaitListEntry    : _LIST_ENTRY [ 0xfffffa80`0418a458 - 0xfffff880`009eb300 ]
+0×0d8 SwapListEntry    : _SINGLE_LIST_ENTRY
+0×0e8 Queue            : 0xfffffa80`03da4bc0 _KQUEUE
+0×0f0 Teb              : 0×000007f6`c41db000 Void
+0×0f8 RelativeTimerBias : 0×00000001`8b165f54
+0×100 Timer            : _KTIMER
+0×140 WaitBlock        : [4] _KWAIT_BLOCK
+0×140 WaitBlockFill4   : [20]  “h???”
+0×154 ContextSwitches  : 0×1817
+0×140 WaitBlockFill5   : [68]  “h???”
+0×184 State            : 0×5 ”
+0×185 NpxState         : 1 ”
+0×186 WaitIrql         : 0 ”
+0×187 WaitMode         : 0 ”
+0×140 WaitBlockFill6   : [116]  “h???”
+0×1b4 WaitTime         : 0xf0172e
+0×140 WaitBlockFill7   : [164]  “h???”
+0×1e4 KernelApcDisable : 0n0
+0×1e6 SpecialApcDisable : 0n0
+0×1e4 CombinedApcDisable : 0
+0×140 WaitBlockFill8   : [40]  “h???”
+0×168 ThreadCounters   : (null)
+0×140 WaitBlockFill9   : [88]  “h???”
+0×198 XStateSave       : (null)
+0×140 WaitBlockFill10  : [136]  “h???”
+0×1c8 Win32Thread      : 0xfffff901`03f206e0 Void
+0×140 WaitBlockFill11  : [176]  “h???”
+0×1f0 Ucb              : (null)
+0×1f8 Uch              : (null)
+0×200 TebMappedLowVa   : (null)
+0×208 QueueListEntry   : _LIST_ENTRY [ 0xfffffa80`02ccf408 - 0xfffffa80`03da4bf0 ]
+0×218 NextProcessor    : 0
+0×21c DeferredProcessor : 1
+0×220 Process          : 0xfffffa80`02cb2940 _KPROCESS
+0×228 UserAffinity     : _GROUP_AFFINITY
+0×228 UserAffinityFill : [10]  “???”
+0×232 PreviousMode     : 1 ”
+0×233 BasePriority     : 8 ”
+0×234 PriorityDecrement : 0 ”
+0×234 ForegroundBoost  : 0y0000
+0×234 UnusualBoost     : 0y0000
+0×235 Preempted        : 0 ”
+0×236 AdjustReason     : 0 ”
+0×237 AdjustIncrement  : 0 ”
+0×238 Affinity         : _GROUP_AFFINITY
+0×238 AffinityFill     : [10]  “???”
+0×242 ApcStateIndex    : 0 ”
+0×243 WaitBlockCount   : 0×1 ”
+0×244 IdealProcessor   : 1
+0×248 ApcStatePointer  : [2] 0xfffffa80`0219c118 _KAPC_STATE
+0×258 SavedApcState    : _KAPC_STATE
+0×258 SavedApcStateFill : [43]  “???”
+0×283 WaitReason       : 0×5 ”
+0×284 SuspendCount     : 0 ”
+0×285 Saturation       : 0 ”
+0×286 SListFaultCount  : 0
+0×288 SchedulerApc     : _KAPC
+0×288 SchedulerApcFill0 : [1]  “??????”
+0×289 ResourceIndex    : 0×1 ”
+0×288 SchedulerApcFill1 : [3]  “???”
+0×28b QuantumReset     : 0×6 ”
+0×288 SchedulerApcFill2 : [4]  “???”
+0×28c KernelTime       : 7
+0×288 SchedulerApcFill3 : [64]  “???”
+0×2c8 WaitPrcb         : (null)
+0×288 SchedulerApcFill4 : [72]  “???”
+0×2d0 LegoData         : (null)
+0×288 SchedulerApcFill5 : [83]  “???”
+0×2db CallbackNestingLevel : 0 ”
+0×2dc UserTime         : 0xa
+0×2e0 SuspendEvent     : _KEVENT
+0×2f8 ThreadListEntry  : _LIST_ENTRY [ 0xfffffa80`01c41378 - 0xfffffa80`01e4edf8 ]
+0×308 MutantListHead   : _LIST_ENTRY [ 0xfffffa80`0219c388 - 0xfffffa80`0219c388 ]
+0×318 ReadOperationCount : 0n392
+0×320 WriteOperationCount : 0n321
+0×328 OtherOperationCount : 0n240
+0×330 ReadTransferCount : 0n1849338
+0×338 WriteTransferCount : 0n1197496
+0×340 OtherTransferCount : 0n4972

This is different when a process is under a debugger and all its threads are frozen except the one that communicates to the debugger like in this case study. In Windows 8 this happens, for example, when we switch to a desktop from IE launched from the start page. Then we would see shortly that iexplore.exe process changes from Running to Suspended in Task Manager Details page. We call this pattern Frozen Process to cover both the new feature and a debugged process case.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Software Diagnostics Services

Friday, July 13th, 2012

For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:

The First Comprehensive Software Diagnostics Service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming 2nd edition of Memory Dump Analysis Anthology, Volume 1

Sunday, April 15th, 2012

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

  • Title: Memory Dump Analysis Anthology, Volume 1
  • Author: Dmitry Vostokov
  • Publisher: OpenTask (Summer 2012)
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 800 pages
  • ISBN-13: 978-1-908043-35-1
  • Hardcover: 800 pages
  • ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Design of Memory Dump Analysis: 7 Steps of Highly Successful Analysts

Monday, February 20th, 2012

I was recently asked by a group of trainees to outline a simple approach to proceed after opening a memory dump. So I came up with these 7 steps:

1. !analyze -v [-hang]
2. Exception (Bugcheck): stack trace analysis with d* and lmv
3. !locks
4. !runaway f (!running)
5. Dump all (processes and) thread stack traces [with 32-bit] ~*kv (!process 0 ff)
6. Search for signs/patterns of abnormal behavior (exceptions, wait chains, message boxes [, from your custom checklist])
7. Narrow analysis down to a specific thread and dump raw stack data if needed [repeat*]

(commands/options in brackets denote kernel/complete dump variation)
[notes in square brackets denote additional options, such as x64 specifics, your product details, etc.]

What are your steps? I would be interested to hear about alternative analysis steps, techniques, etc.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

New Book: Advanced Windows Memory Dump Analysis

Friday, January 27th, 2012

Advanced training sessions time may not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

  • Title: Advanced Windows Memory Dump Analysis with Data Structures: Training Course Transcript and WinDbg Practice Exercises with Notes
  • Description: The full transcript of Memory Dump Analysis Services Training with 10 step-by-step exercises, notes, and selected Q&A.
  • Authors: Dmitry Vostokov, Memory Dump Analysis Services
  • Publisher: OpenTask (January 2012)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 180 pages
  • ISBN-13: 978-1908043344

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

2 WinDbg Scripts That Changed The World

Saturday, December 3rd, 2011

If not for you then definitely for me because I now diagnose Spiking Thread pattern much faster. One of the readers if this blog asked me whether there is !runaway command equivalent for kernel and complete memory dumps. So, after some thinking I gave it a try especially in the context of WinDbg scripting exercises designed for Advanced Windows Memory Dump Analysis training. As a result I wrote 2 scripts initially that you can try yourself. Their output here is taken from a complete memory dump I used for Fundamentals of Complete Crash and Hang Memory Dump Analysis presentation.

The first one dumps the most CPU consuming threads for user and kernel mode:

$$
$$ krunawaymost.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest UserTime value: ”
? @$t0
!thread @$t2 ff
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest KernelTime value: ”
? @$t0
!thread @$t2 ff

0: kd> $$><c:\Scripts\krunawaymost.wds
The largest UserTime value:
Evaluate expression: 5470 = 00000000`0000155e

THREAD fffffa800451d720  Cid 1418.17fc  Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa800442ab30       Image:         ApplicationE.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22295          Ticks: 0
Context Switch Count      27960            
UserTime                  00:01:25.332
KernelTime                00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3

The largest KernelTime value:
Evaluate expression: 187 = 00000000`000000bb

THREAD fffffa80098d7b60  Cid 07bc.0a14  Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
    fffffa8008a4a030  NotificationEvent
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa80096beb30       Image:         dwm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22294          Ticks: 1 (0:00:00:00.015)
Context Switch Count      15473                 LargeStack
UserTime                  00:00:06.801
KernelTime                00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The second script takes two arguments and shows all threads that have UserTime and KernelTime ticks value greater than (you can have the idea of the maximum from the previous script):

$$
$$ krunawaygt.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); r $t0 = $arg1; .if (@$t1 > @$t0) {!thread @#Thread ff}”
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); r $t0 = $arg2; .if (@$t1 > @$t0) {!thread @#Thread ff}”

Using hints from the previous script run (the largest UserTime ticks value is 0×155e) we now get threads that spent more than 0×100 ticks in user mode:

0: kd> $$>a<c:\Scripts\krunawaygt.wds 100 100
THREAD fffffa800843e060  Cid 03f4.0658  Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa800843c2c0  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      21211          Ticks: 1084 (0:00:00:16.910)
Context Switch Count      6028            
UserTime                  00:00:10.140
KernelTime                00:00:00.296
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d4bc70 Current fffff88009d4b660
Base fffff88009d4c000 Limit fffff88009d46000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mprtp.dll -
Child-SP          RetAddr           Call Site
fffff880`09d4b6a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d4b7e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d4b870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d4b930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d4b9c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d4ba70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d4bae0)
00000000`0209fb08 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`0209fb10 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`0209fb70 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`0209fbb0 00000000`0045cbc0 mprtp!MpPluginSignatureChange+0×3e170
00000000`0209fbb8 000007fe`fbac25ff 0×45cbc0
00000000`0209fbc0 00000000`00466610 FLTLIB!FilterGetMessage+0×2b
00000000`0209fc20 00000000`00000000 0×466610

THREAD fffffa800845c060  Cid 03f4.065c  Teb: 000007fffff8e000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa800843c2c0  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      21520          Ticks: 775 (0:00:00:12.090)
Context Switch Count      4979            
UserTime                  00:00:04.149
KernelTime                00:00:00.156
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d52c70 Current fffff88009d52660
Base fffff88009d53000 Limit fffff88009d4d000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mprtp.dll -
Child-SP          RetAddr           Call Site
fffff880`09d526a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d527e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d52870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d52930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d529c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d52a70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d52ae0)
00000000`01ccf498 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`01ccf4a0 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`01ccf500 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`01ccf540 00000000`0045d030 mprtp!MpPluginSignatureChange+0×3e170
00000000`01ccf548 000007fe`fbac25ff 0×45d030
00000000`01ccf550 00000000`004666b0 FLTLIB!FilterGetMessage+0×2b
00000000`01ccf5b0 00000000`00000000 0×4666b0

THREAD fffffa80092b7060  Cid 03f4.1268  Teb: 000007fffff6a000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Alertable
    fffffa8009299140  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      7762           Ticks: 14533 (0:00:03:46.716)
Context Switch Count      3297            
UserTime                  00:00:06.489
KernelTime                00:00:00.499
Win32 Start Address ntdll!TppWorkerThread (0×00000000772bfbc0)
Stack Init fffff8800e620c70 Current fffff8800e620680
Base fffff8800e621000 Limit fffff8800e61b000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0e6206c0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0e620800 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`0e620890 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`0e620950 fffff800`02ecdb36 nt!IoRemoveIoCompletion+0×47
fffff880`0e6209e0 fffff800`02ee0ed3 nt!NtWaitForWorkViaWorkerFactory+0×285
fffff880`0e620ae0 00000000`772f2c1a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0e620ae0)
00000000`0540f998 00000000`772bfe0b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0540f9a0 00000000`7719652d ntdll!TppWorkerThread+0×2c9
00000000`0540fca0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0540fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa80098d7b60  Cid 07bc.0a14  Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
    fffffa8008a4a030  NotificationEvent
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa80096beb30       Image:         dwm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22294          Ticks: 1 (0:00:00:00.015)
Context Switch Count      15473                 LargeStack
UserTime                  00:00:06.801
KernelTime                00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa800451d720  Cid 1418.17fc  Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa800442ab30       Image:         ApplicationE.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22295          Ticks: 0
Context Switch Count      27960            
UserTime                  00:01:25.332
KernelTime                00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3

Memory Dump Analysis Services is now working to incorporate client-side WinDbg scripting into their CARE2 architecture. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

Forthcoming Volume 6 of Memory Dump Analysis Anthology

Thursday, November 3rd, 2011

The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:

  • 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
  • 4 new pattern interaction case studies
  • 11 new trace analysis patterns
  • New Debugware pattern
  • Introduction to UI problem analysis patterns
  • Introduction to intelligence analysis patterns
  • Introduction to unified debugging pattern language
  • Introduction to generative debugging, metadefect template library and DNA of software behaviour
  • The new school of debugging and trends
  • .NET memory dump analysis checklist
  • Software trace analysis checklist
  • Introduction to close and deconstructive readings of a software trace
  • Memory dump analysis compass
  • Computical and Stack Trace Art
  • The abductive reasoning of Philip Marlowe
  • Orbifold memory space and cloud computing
  • Memory worldview
  • Interpretation of cyberspace
  • Relationship of memory dumps to religion
  • Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 6
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 300 pages
  • Publisher: Opentask (December 2011)
  • ISBN-13: 978-1-908043-19-1
  • Hardcover: 300 pages
  • Publisher: Opentask (January 2012)
  • ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Music for Debugging: Navigating the Long List

Monday, September 12th, 2011

If you like the completeness, grand complete memory dumps, multi-volume oeuvres (the more volumes the better) and natural memory visualization you would then like to open this box and listen to this complete performance achievement to get energy and inspiration for long debugging sessions:

Liszt: The Complete Piano Music

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Advanced Windows Memory Dump Analysis with Data Structures Training Course

Sunday, August 14th, 2011

Due to the need to extend existing basic and intermediate Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises advanced training course. Here is the description and registration information:

Learn how to navigate through memory dump space and Windows data structures to troubleshoot and debug complex software incidents. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose structural and behavioral patterns in 32-bit and 64-bit process, kernel and complete memory dumps.

Advanced Windows Memory Dump Analysis Logo

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

  1. A full transcript in PDF format (retail price $200)
  2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
  3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic and intermediate level Windows memory dump analysis: ability to list processors, processes, threads, modules, apply symbols, walk through stack traces and raw stack data, diagnose patterns such as heap corruption, CPU spike, memory and handle leaks, access violation, stack overflow, critical section and resource wait chains and deadlocks. If you don’t feel comfortable with prerequisites then Accelerated Windows Memory Dump Analysis training is recommended to take (or purchase a corresponding book) before attending this course.

Audience: Software developers, software technical support and escalation engineers.

Session: December 9, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/3788047691824598784

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -