Software Diagnostics Library

Today we introduce an icon for Multiple Exceptions (managed space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This month marks 5 years of blogging that resulted in more than 2,500 posts across 8 blogs. I would like to thank everyone for their continuing support! The major milestones are marked on the updated timeline.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Abscess - A Bad Service and Customer Experience Short Story. Can also be an abbreviation for a bad support …

Examples: After combating an abscess a customer decided to take a social media approach. An abscess process.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services announces development of the first memory dump analysis certification and is looking for volunteers to participate in its beta program. Please visit its website for further details.

Source: http://www.dumpanalysis.com/anon-beta-exam-mda-bi-w

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to the need to extend existing basic and intermediate Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises advanced training course. Here is the description and registration information:

Learn how to navigate through memory dump space and Windows data structures to troubleshoot and debug complex software incidents. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose structural and behavioral patterns in 32-bit and 64-bit process, kernel and complete memory dumps.

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

1. A full transcript in PDF format (retail price $200)
2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic and intermediate level Windows memory dump analysis: ability to list processors, processes, threads, modules, apply symbols, walk through stack traces and raw stack data, diagnose patterns such as heap corruption, CPU spike, memory and handle leaks, access violation, stack overflow, critical section and resource wait chains and deadlocks. If you don’t feel comfortable with prerequisites then Accelerated Windows Memory Dump Analysis training is recommended to take (or purchase a corresponding book) before attending this course.

Audience: Software developers, software technical support and escalation engineers.

Session: December 9, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/3788047691824598784

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally you can even learn a WinDbg command from a certificate. Memory Dump Analysis Services has created a certificate with dv WinDbg command on it:

Source: http://www.dumpanalysis.com/sample-certificate

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to popular demand and the need to extend existing Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises the new training course. Here is the description and registration information:

Learn how to analyze .NET application and service crashes and freezes, navigate through memory dump space (managed and unmanaged code) and diagnose corruption, leaks, CPU spikes, blocked threads, deadlocks, wait chains, resource contention, and much more. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose patterns in 32-bit and 64-bit process memory dumps.

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

1. A full transcript in PDF format (retail price $200)
2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic .NET programming and debugging.

Audience: Software developers, software technical support and escalation engineers.

Session: October 28, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/423991811034037760

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Recently when looking at one of process memory dumps with CLR runtime I got this message when trying to load SOS extension from the appropriate Framework location:

0:022> .load [..]\Microsoft.NET\Framework\v2.0.50727\sos.dll
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------

So I literally did what was asked by saving a new dump from the dump:

0:022> .dump /ma c:\userdumps\dump-as-asked.dmp
Creating c:\userdumps\dump-as-asked.dmp - mini user dump
Dump successfully written

I got a slightly bigger dump than the original and after that SOS didn’t complain when I opened the new dump and loaded the extension.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I showed this artwork to many people and they responded that they didn’t understand or hope one day they would understand. Only one guy (a professional manager) responded positively with understanding. What I think is that real art can be interpreted in many ways. So I kindly await your criticism.

PS. This Computicart (Computical Art) work was inspired by a pet parrot at home. I’ve been observing its behaviour for more than 6 months and tried to discern a few patterns. Its name is KiKi (not related to Ki* functions).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After reading many history books I found that the real historical, sociological and psychological force (often seen unconscious from outside) is the will to be memorized (saved in memories, in memory dumps, also called as the will to be remembered) and not the will to power (Nietzsche). As often with the will it is also linked with vices and virtues. For the vice side we site Michael Burleigh: “Concern with posterity was apparent among the Nazi leadership, […]” and what we see now “… B-movie villains were self-consciously assigning themselves parts within an A-movie which runs and runs” today (documentaries, books, magazines, TV and the so called ‘Hitler industry’ “by the way of continuity”). As a more recent example, the recent tragedy in Norway clearly shows the desire of the protagonist to be memorized (masqueraded as a political agenda).

This is not to say that the will to power is not important, it is important by the way that by being memorized you actually exert more power in the future. Memorianity (memory religion) and its philosophical foundation (memoidealism) promote the will to be memorized as a virtue if all your deeds are seen as a virtue (see Memorianity and Morality).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

During the previous several months many people expressed their interest in the training (the next one is scheduled for November) but its time was not suitable due to the very different geographic time zones. So I have decided to publish this training in book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

• Title: Accelerated Windows Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
• Description: The full transcript of Memory Dump Analysis Services Training with 21 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers about 50 crash dump analysis patterns from process, kernel and complete memory dumps.
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Publisher: OpenTask (August 2011)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 360 pages
• ISBN-13: 978-1908043290

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to popular demand (the previous training was fully booked) Memory Dump Analysis Services scheduled the next training sessions.

Learn how to analyze application, service and system crashes and freezes, navigate through memory dump space and diagnose heap corruption, memory leaks, CPU spikes, blocked threads, deadlocks, wait chains, and much more. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of more than 20 practical step-by-step exercises using WinDbg highlighting more than 50 patterns diagnosed in 32-bit and 64-bit process, kernel and complete memory dumps.

Public preview (selected slides) of the previous training

Memory Dump Analysis Services organizes a training course.

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of 4 two-hour sessions (2 hours every day). When you finish the training you additionally get:

1. A full transcript in PDF format (retail price $200)
2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic Windows troubleshooting

Session 1: November 1, 2011 4:00 PM - 6:00 PM GMT
Session 2: November 2, 2011 4:00 PM - 6:00 PM GMT
Session 3: November 3, 2011 4:00 PM - 6:00 PM GMT
Session 4: November 4, 2011 4:00 PM - 6:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now.

If scheduled dates or time are not suitable for you Memory Dump Analysis Services offers the same training in book format. 

Training testimonials:

I would like to thank you and recommend your training. I think that the “Accelerated Windows Memory Dump Analysis” training is a pin-point, well taught training. I think it’s the leading training in the dump analysis area and I’ve enjoyed it, the books and materials are very detailed and well written and Dmitry answered all of the needed question. In addition after the training Dmitry sent a PDF with written answers and more information about the questions that were asked. I will give this training 5/5. Thank you Dmitry.

Yaniv Miron, Security Researcher, IL.Hack 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finction (or simply Finc) - A function with a flaw, having abnormal behaviour, a candidate for a fix or axing.

Examples: After delivering a hotfix with axed code they were embarrassed to see that finction again on a stack trace.

Etymology: Derives from “function”. Inspired by MDAA V3 errata.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Never thought that I would one day bugtate Bill Gates but found his quote in Knuth’s The Art of Computer Programming, Volume 1:

Memory Dumps “have” not “changed in the past two decades.”

Bill Gates

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

UI Message pattern is very useful for troubleshooting system-wide issues because we can map visual behaviour to various activity regions and consider such messages as significant events.

#    Module  PID  TID  Time         Message
[...]
2782 ModuleA 2124 5648 10:58:03.356 CreateWindow: Title "..." Class "..."
[...]
3512 ModuleA 2124 5648 10:58:08.154 Menu command: Save Data
[...]
3583 ModuleA 2124 5648 10:58:08.155 CreateWindow: Title "Save As" Class "Dialog"
[... Data update and replication related messages ...]
4483 ModuleA 2124 5648 10:58:12.342 DestroyWindow: Title "Save As" Class "Dialog"
[...]

By filtering the emitting module we can create an adjoint thread:

#    Module  PID  TID  Time         Message
[...]
2782 ModuleA 2124 5648 10:58:03.356 CreateWindow: Title "..." Class "..."
3512 ModuleA 2124 5648 10:58:08.154 Menu command: Save Data
3583 ModuleA 2124 5648 10:58:08.155 CreateWindow: Title "Save As" Class "Dialog"
4483 ModuleA 2124 5648 10:58:12.342 DestroyWindow: Title "Save As" Class "Dialog"
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Second Eye (or sometimes a stronger variant “second pair of eyes”) - another engineer you typically need when you don’t see anything useful in a memory dump, software trace or source code for problem resolution purposes. You are anxious to recommend something useful.

Examples: Don’t see anything in this huge trace. I need a second eye.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Recently analyzed a process memory dump and noticed that it (up and running) survived system reboot

0:000> version
Windows Vista Version 6000 MP (2 procs) Free x64
Product: WinNt, suite: SingleUserTS Personal
kernel32.dll version: 6.0.6000.16386 (vista_rtm.061101-2205)
Machine Name:
Debug session time: Tue Jul 12 16:53:07.000 2011 (UTC + 1:00)
System Uptime: 0 days 1:27:04.516
Process Uptime: 1 days 4:05:35.000
  Kernel time: 0 days 0:00:13.000
  User time: 0 days 0:00:04.000
[…]

I have a hypothesis how this could have happened. Interested in knowing yours. I’ll write mine later on.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that I had a visitor from Vatican City (as reported by Google Analytics):

I hope they were interested in Memory Religion (Memorianity) where I have the title of Memoriarch.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It came to my attention that almost every huge or not so x64 kernel or complete memory dump is diagnosed with excessive pool usage. Sometimes it is too excessive like in the following example:

0: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:     8387414 (  33549656 Kb)
 Page File: \??\D:\pagefile.sys
   Current:  33856856 Kb  Free Space:  33855520 Kb
   Minimum:  33856856 Kb  Maximum:     46364420 Kb
 Available Pages:     7231844 (  28927376 Kb)
 ResAvail Pages:      7763458 (  31053832 Kb)
 Locked IO Pages:           0 (         0 Kb)
 Free System PTEs:   33556220 ( 134224880 Kb)
 Modified Pages:         2759 (     11036 Kb)
 Modified PF Pages:      2759 (     11036 Kb)
 NonPagedPool Usage: 650867425 (2603469700 Kb)
 NonPagedPoolNx Usage:  83715 (    334860 Kb)
 NonPagedPool Max:    6271754 (  25087016 Kb)
 ********** Excessive NonPaged Pool Usage *****
 PagedPool 0 Usage:     48923 (    195692 Kb)
 PagedPool 1 Usage:     39797 (    159188 Kb)
 PagedPool 2 Usage:     37412 (    149648 Kb)
 PagedPool 3 Usage:     37536 (    150144 Kb)
 PagedPool 4 Usage:     37453 (    149812 Kb)
 PagedPool Usage:      201121 (    804484 Kb)
 PagedPool Maximum:  33554432 ( 134217728 Kb)
 Session Commit:        15829 (     63316 Kb)
 Shared Commit:          7198 (     28792 Kb)
 Special Pool:              0 (         0 Kb)
 Shared Process:       158498 (    633992 Kb)
 PagedPool Commit:     201147 (    804588 Kb)
 Driver Commit:          5761 (     23044 Kb)
 Committed pages:     1126203 (   4504812 Kb)
 Commit limit:       16851145 (  67404580 Kb)

What we can see above is that the amount of used nonpaged pool is more than 2.5 Tb which is far less than the amount of physical memory + page file size (both in total do not exceed 100 Gb). So I conclude that Windows architects did the impossible and are able to create information (pages) from vacuum like matter can be created from vacuum fluctuations. Perhaps they are a step closer to implement some features from Cantor OS.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One way to quickly check for something suspicious in a memory dump is to convert it to a debugger log (for example, stack trace collection) and search for textual strings such as “Waiting for“, “Terminate“, “Stop”, “Mutant“, “Exception“, “Crit“, “MessageBox“, “SuspendCount“, etc. The vocabulary, of course, is OS dependent, can have false positives, and can change over time. We name this pattern Problem Vocabulary. It is similar to Vocabulary Index software trace analysis pattern.

For example, recently in a complete memory dump involving lots of ALPC wait chains with potential inter-process deadlock we found the following thread having long waiting time (exceeding ALPC threads waiting times) pointing to a process object to examine further:

THREAD fffffa801338b950  Cid 02a0.7498  Teb: 000007fffffd8000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8012a39b30  ProcessObject
Not impersonating
DeviceMap                 fffff8a000008a70
Owning Process            fffffa800a31d040       Image:         smss.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      9752080        Ticks: 5334204 (0:23:09:06.937)
Context Switch Count      38            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0×000000007722fbc0)
Stack Init fffff88020259db0 Current fffff88020259900
Base fffff8802025a000 Limit fffff88020254000 Call 0
Priority 11 BasePriority 11 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`20259940 fffff800`01693f92 nt!KiSwapContext+0×7a
fffff880`20259a80 fffff800`016967af nt!KiCommitThreadWait+0×1d2
fffff880`20259b10 fffff800`01984b2e nt!KeWaitForSingleObject+0×19f
fffff880`20259bb0 fffff800`0168df93 nt!NtWaitForSingleObject+0xde
fffff880`20259c20 00000000`7726135a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`20259c20)
00000000`0048f648 00000000`48026517 ntdll!NtWaitForSingleObject+0xa
00000000`0048f650 00000000`480269c4 smss!SmpTerminateCSR+0xa3
00000000`0048f6a0 00000000`48023670 smss!SmpStopCsr+0×44
00000000`0048f6d0 00000000`77288137 smss!SmpApiCallback+0×338
00000000`0048f900 00000000`7722feff ntdll! ?? ::FNODOBFM::`string’+0×1f718
00000000`0048f990 00000000`77274a00 ntdll!TppWorkerThread+0×3f8
00000000`0048fc90 00000000`00000000 ntdll!RtlUserThreadStart+0×25

In that process we could see a blocking module and recommended to contact its vendor: 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -