Software Diagnostics Library

Monthly summary of my Management Bits and Tips blog including January posts:

Management Bit and Tip 0×2000

Management Bit and Tip 0×4000

A Thread Was Killed 

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Literate Scientist blog including January reviews:

Blog Anniversary

Social Sciences as Sorcery

Literate Scientists and Their Books

Reality Rules

A Brief History of Theology

- Dmitry Vostokov @ DumpAnalysis.org -

A contribution to Software Resistentialism:

Software objects can be classified scientifically into three major categories: those that don’t work, those that crash and those that hang.

Russell Wayne Baker

- Dmitry Vostokov @ DumpAnalysis.org -

“Don’t” guess “it, get a larger” dump.

Anthony’s Law of Force

- Dmitry Vostokov @ DumpAnalysis.org -

Debugger logs (textual output) from commands like !process 0 ff and various scripts can be very long and consist of thousands of pages. I found the following reading technique useful for my daily memory dump analysis activities:

CSA-QSA

Checklists-Skim-Analyze—Questions-Survey-Analyze   

1. First, have a checklist

2. Skim through the log several times

3. Write analysis notes

4. Have a list of questions based on problem description and steps 1-3

5. Survey the log

6. Write analysis notes

Repeat steps 2,3 and 5,6 if necessary.

This technique can also be applied to reading any large logs, for example, voluminous CDF or ETW traces.

- Dmitry Vostokov @ DumpAnalysis.org -

Named after Georg Cantor CAN.TOR.OS^∞ brings computation from the distant future into today. The transfinite worldview and universe of tomorrow into the finite worldview and universe of today. Cantor OS drives transfinite computing and saves transfinite memory dumps. More on this in subsequent parts as I have to come back to finite memory dumps… One cautious note though: transfinite doesn’t mean absolute infinity, or God-like computation, the latter is the realm of Memory Religion. 

^(∞) TOR is a new transfinite operation in addition to finite OR, AND or XOR 

- Dmitry Vostokov @ DumpAnalysis.org -

These dumps are larger than any finite memory dump and contain all of them inside (see the definition of a transfinite number). Think about them as a variant of the Library of Babel where all possible memory snapshots of your Windows or Linux PC are stored including Googol dumps. If you have some code then all possible code defects are there too. An interesting question then arises. If this dump is collected what kind of patterns we can see there? Are these patterns extrapolated infinite versions of finite patterns or there come new ones specific to transfinite computations? More on this in the next parts.

- Dmitry Vostokov @ DumpAnalysis.org -

The first issue of Debugged! MZ/PE magazine is going to be printed on debugged paper (not to confused with common bugs-free paper used in publishing houses and printing factories). Once you open the first issue you would instantly recognize that!

- Dmitry Vostokov @ DumpAnalysis.org -

Looked this evening at Amazon and found that the book achieved #1 status (although it might not be the case at the time when you are reading this post):

- Dmitry Vostokov @ DumpAnalysis.org -

Named after Bernhard Riemann, this programming language gives software defects first-class status as alternative branches of computation, comparable with multivalued functions and Riemann surfaces. Bugs become first-class constructs. It is reflected in the language syntax, semantics and pragmatics. More on this later.

- Dmitry Vostokov @ DumpAnalysis.org -

The main problem of memory dump analysis is the lack of consistent kernel virtual memory dumps saved on demand without system halt. LiveKd and Win32DD tools are physical memory dumpers only and do not save kernel memory dump files. These dumps are known to be inconsistent and I elaborated on different schemes to save memory consistently, for example, 1) to partition physical memory into 2 parts from OS boot time, 2) when memory snapshot is needed raise IRQL on all processors, 3) pump memory contents from one part to another (with compression if necessary, in such partition the reserved part of physical memory could be smaller), 4) lower IRQL on all processors to resume normal OS functions and 5) save consistent memory snapshot from reserved part of physical memory to a dump file in the background. The crucial feature of osmosis is its bipartite division and membrane. Hence the name of the project: 

OSMOSIS

Optimally Saved Memory of System Internal State

Optimally Saved Memory (of) Operating System Internal State

This is, of course, for OS running on physical machines, virtual machine case is much simpler in theory because we can freeze the whole VM or save its snapshot and later run an external tool or file converter on it.

- Dmitry Vostokov @ DumpAnalysis.org -

One of the good outcomes of the previously announced restructuring: the book Crash Dump Analysis for System Administrators (Windows edition) has been prioritized to be published on 30th of November, 2009 due to the overwhelming demand. The book will soon be available for pre-orders.

- Dmitry Vostokov @ DumpAnalysis.org -

One analysis problem that happens frequently is the absence of stack traces due to kernel stack pages being paged out and therefore not present in a complete memory dump that only contains physical memory. This shouldn’t be a problem for kernel or process memory dumps because they contain virtual memory. The problem usually manifests itself either on busy systems utilizing almost all physical memory or on over-aged systems where certain processes hadn’t been used for a long time. It could also be the case when a problem happened some time ago and only diagnosed much later. For example, this LPC wait chain for coupled processes happened to be 2 days ago before the dump was saved:

0: kd> !thread fffffadfcf9e8bf0 1f
THREAD fffffadfcf9e8bf0  Cid 61f0.2c70  Teb: 000007fffffdd000 Win32Thread: fffff97ff381a480 WAIT: (Unknown) UserMode Non-Alertable
    fffffadfcf9e8f58  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 01e2cb39:
Current LPC port fffffa800e5a9d10
Impersonation token:  fffffa80039cd050 (Level Impersonation)
Owning Process fffffadffc7c7c20      
Image:         applicationA.exe
Wait Start TickCount      12018444           
Ticks: 11312740 (2:01:06:01.562)
Context Switch Count      456                 LargeStack
UserTime                  00:00:00.046
KernelTime                00:00:00.078
Start Address applicationA (0×0000000100061411)
Stack Init fffffadc125d4e00 Current fffffadc125d48e0
Base fffffadc125d5000 Limit fffffadc125cc000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0
Kernel stack not resident.

0: kd> !lpc message 01e2cb39
Searching message 1e2cb39 in threads …
    Server thread fffffadff93c5bf0 is working on message 1e2cb39

0: kd> !thread fffffadff93c5bf0 1f
THREAD fffffadff93c5bf0  Cid 0218.5130  Teb: 000007ffffcbc000 Win32Thread: 0000000000000000 WAIT: (Unknown) UserMode Non-Alertable
    fffffadff6c71c70  SynchronizationEvent
Impersonation token:  fffffa803bde5060 (Level Impersonation)
Owning Process fffffadcde439280     
Image:         applicationB.exe
Wait Start TickCount      12018444        
Ticks: 11312740 (2:01:06:01.562)
Context Switch Count      12            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×0000000001e2cb39
LPC Server thread working on message Id 1e2cb39
Start Address kernel32 (0×0000000077d6b6a0)
Stack Init fffffadc28b19e00 Current fffffadc28b19950
Base fffffadc28b1a000 Limit fffffadc28b14000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Kernel stack not resident.

One of tricks I started to recommend in such cases is to save user dumps of processes that could possibly be paged out before forcing a complete memory dump but I haven’t yet got enough data to see whether this helps or not.

- Dmitry Vostokov @ DumpAnalysis.org -

I found today that I never came up with a definition for a memory dump analysis pattern and only mentioned similar design pattern definition in the first pattern post. Now it is time to draft it:

a common recurrent identifiable problem together with a set of recommendations and possible solutions to apply in a specific context

In memory dump analysis pattern catalogue some identified patterns are left without a set of recommendations and I’m going to rectify this in the nearest future.

- Dmitry Vostokov @ DumpAnalysis.org -

While preparing a presentation for an engineering conference I found that a page to reference various wait chain patterns is necessary, so I created this post:

• Wait Chain (general)
• Wait Chain (critical sections)
• Wait Chain (executive resources)
• Wait Chain (thread objects)
• Wait Chain (LPC/ALPC)
• Wait Chain (process objects)
• Wait Chain (RPC)
• Wait Chain (window messaging)
• Wait Chain (named pipes)
• Wait Chain (mutex objects)
• Wait Chain (pushlocks)
• Distributed Wait Chain
• Wait Chain (CLR monitors)
• Screwbolt Wait Chain
• Wait Chain (RTL_RESOURCE)
• Wait Chain (modules)
• Wait Chain (nonstandard synchronization)
• Wait Chain (C++11, condition variable)
• Wait Chain (SRW lock)

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

While preparing a presentation for an engineering conference I found that another different page to reference link time problems is necessary, so I created this post:

• Missing Component (general)
• Missing Component (static linking, user mode)
• Duplicated Module
• Unloaded Module

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

While preparing a presentation for an engineering conference I found that yet another page to reference different kinds of insufficient memory patterns is necessary, so I created this post:

• Insufficient Memory (committed memory)
• Insufficient Memory (handle leak)
• Insufficient Memory (kernel pool)
• Insufficient Memory (PTE)
• Insufficient Memory (module fragmentation)
• Insufficient Memory (physical memory)
• Insufficient Memory (control blocks)
• Insufficient Memory (reserved virtual memory)
• Memory Leak (process heap)
• Memory Leak (.NET heap)
• Memory Leak (page tables)
• Memory Leak (I/O completion packets)
• Punctuated Memory Leak
• Insufficient Memory (session pool)
• Memory Fluctuation (process heap)
• Insufficient Memory (stack trace database)
• Insufficient Memory (region)
• Memory Leak (regions)
• Handle Leak
• Reference Leak
• Object Distribution Anomaly (.NET heap)
• Handle Limit (GDI, kernel space)
• Handle Limit (GDI, user space)
• Relative Memory Leak
• Insufficient Memory (stack)

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

While preparing a presentation for an engineering conference I found that another page to reference different kinds of dynamic memory corruption patterns is necessary, so I created this post too:

• Dynamic Memory Corruption (process heap)
• Dynamic Memory Corruption (kernel pool)
• Double Free (process heap)
• Double Free (kernel pool)
• Dynamic Memory Corruption (managed heap)

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

While preparing a presentation for an engineering conference I found that a page to reference all different kinds of deadlocks is necessary, so I created this post:

• Deadlock (critical sections)
• Deadlock (executive resources)
• Deadlock (mixed objects, user space)
• Deadlock (LPC)
• Deadlock (mixed objects, kernel space)
• Deadlock (self)
• Livelock
• Deadlock (managed space)
• Deadlock (.NET Finalizer)

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’ve always wondered why LockCount is sometimes negative in the latest OS versions since SPs for Windows Server 2003:

0:000> !locks

CritSec +319ee0 at 00319EE0
LockCount -90
RecursionCount 1
OwningThread 1198
EntryCount 0
ContentionCount 734a6
*** Locked

CritSec +202e3d0 at 0202E3D0
LockCount -2
RecursionCount 1
OwningThread 1198
EntryCount 0
ContentionCount 0
*** Locked

CritSec +1b69ad4 at 01B69AD4
LockCount -2
RecursionCount 1
OwningThread 1198
EntryCount 0
ContentionCount 0
*** Locked

Finally the post on NT Debugging Blog pointed to WinDbg Help article (the section Interpreting Critical Section Fields in Windows Server 2003 SP1 and Later):

http://msdn.microsoft.com/en-us/library/cc267476.aspx

Be aware though that negative values could also mean Critical Section Corruption pattern:

CritSec printerA!DlgProc+53b38 at 107cddd8
WaiterWoken Yes
LockCount -69154612
RecursionCount 276618309
OwningThread 107cdd36
EntryCount 0
ContentionCount 0
*** Locked 

- Dmitry Vostokov @ DumpAnalysis.org -