Software Diagnostics Library

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Diagnostic Policy Service, DPS (pp. 330 - 331)

SMART (p. 332) - Don’t confuse with recursive acronym Smart Memory Analysis in Real Time (coined by me)

Windows system responsiveness performance diagnostics (p. 332)

Program Compatibility Assistant, PCA (p. 333)

_EPROCESS and _KPROCESS (pp. 337 - 339) - x64 equivalents from W2K8:

lkd> dt _EPROCESS
ntdll!_EPROCESS
+0x000 Pcb              : _KPROCESS
+0x0c0 ProcessLock      : _EX_PUSH_LOCK
+0x0c8 CreateTime       : _LARGE_INTEGER
+0x0d0 ExitTime         : _LARGE_INTEGER
+0x0d8 RundownProtect   : _EX_RUNDOWN_REF
+0x0e0 UniqueProcessId  : Ptr64 Void
+0x0e8 ActiveProcessLinks : _LIST_ENTRY
+0x0f8 QuotaUsage       : [3] Uint8B
+0x110 QuotaPeak        : [3] Uint8B
+0x128 CommitCharge     : Uint8B
+0x130 PeakVirtualSize  : Uint8B
+0x138 VirtualSize      : Uint8B
+0x140 SessionProcessLinks : _LIST_ENTRY
+0x150 DebugPort        : Ptr64 Void
+0x158 ExceptionPortData : Ptr64 Void
+0x158 ExceptionPortValue : Uint8B
+0x158 ExceptionPortState : Pos 0, 3 Bits
+0x160 ObjectTable      : Ptr64 _HANDLE_TABLE
+0x168 Token            : _EX_FAST_REF
+0x170 WorkingSetPage   : Uint8B
+0x178 AddressCreationLock : _EX_PUSH_LOCK
+0x180 RotateInProgress : Ptr64 _ETHREAD
+0x188 ForkInProgress   : Ptr64 _ETHREAD
+0x190 HardwareTrigger  : Uint8B
+0x198 PhysicalVadRoot  : Ptr64 _MM_AVL_TABLE
+0x1a0 CloneRoot        : Ptr64 Void
+0x1a8 NumberOfPrivatePages : Uint8B
+0x1b0 NumberOfLockedPages : Uint8B
+0x1b8 Win32Process     : Ptr64 Void
+0x1c0 Job              : Ptr64 _EJOB
+0x1c8 SectionObject    : Ptr64 Void
+0x1d0 SectionBaseAddress : Ptr64 Void
+0x1d8 QuotaBlock       : Ptr64 _EPROCESS_QUOTA_BLOCK
+0x1e0 WorkingSetWatch  : Ptr64 _PAGEFAULT_HISTORY
+0x1e8 Win32WindowStation : Ptr64 Void
+0x1f0 InheritedFromUniqueProcessId : Ptr64 Void
+0x1f8 LdtInformation   : Ptr64 Void
+0x200 Spare            : Ptr64 Void
+0x208 VdmObjects       : Ptr64 Void
+0x210 DeviceMap        : Ptr64 Void
+0x218 EtwDataSource    : Ptr64 Void
+0x220 FreeTebHint      : Ptr64 Void
+0x228 PageDirectoryPte : _HARDWARE_PTE
+0x228 Filler           : Uint8B
+0x230 Session          : Ptr64 Void
+0x238 ImageFileName    : [16] UChar
+0x248 JobLinks         : _LIST_ENTRY
+0x258 LockedPagesList  : Ptr64 Void
+0x260 ThreadListHead   : _LIST_ENTRY
+0x270 SecurityPort     : Ptr64 Void
+0x278 Wow64Process     : Ptr64 Void
+0x280 ActiveThreads    : Uint4B
+0x284 ImagePathHash    : Uint4B
+0x288 DefaultHardErrorProcessing : Uint4B
+0x28c LastThreadExitStatus : Int4B
+0x290 Peb              : Ptr64 _PEB
+0x298 PrefetchTrace    : _EX_FAST_REF
+0x2a0 ReadOperationCount : _LARGE_INTEGER
+0x2a8 WriteOperationCount : _LARGE_INTEGER
+0x2b0 OtherOperationCount : _LARGE_INTEGER
+0x2b8 ReadTransferCount : _LARGE_INTEGER
+0x2c0 WriteTransferCount : _LARGE_INTEGER
+0x2c8 OtherTransferCount : _LARGE_INTEGER
+0x2d0 CommitChargeLimit : Uint8B
+0x2d8 CommitChargePeak : Uint8B
+0x2e0 AweInfo          : Ptr64 Void
+0x2e8 SeAuditProcessCreationInfo : _SE_AUDIT_PROCESS_CREATION_INFO
+0x2f0 Vm               : _MMSUPPORT
+0x358 MmProcessLinks   : _LIST_ENTRY
+0x368 ModifiedPageCount : Uint4B
+0x36c Flags2           : Uint4B
+0x36c JobNotReallyActive : Pos 0, 1 Bit
+0x36c AccountingFolded : Pos 1, 1 Bit
+0x36c NewProcessReported : Pos 2, 1 Bit
+0x36c ExitProcessReported : Pos 3, 1 Bit
+0x36c ReportCommitChanges : Pos 4, 1 Bit
+0x36c LastReportMemory : Pos 5, 1 Bit
+0x36c ReportPhysicalPageChanges : Pos 6, 1 Bit
+0x36c HandleTableRundown : Pos 7, 1 Bit
+0x36c NeedsHandleRundown : Pos 8, 1 Bit
+0x36c RefTraceEnabled  : Pos 9, 1 Bit
+0x36c NumaAware        : Pos 10, 1 Bit
+0x36c ProtectedProcess : Pos 11, 1 Bit
+0x36c DefaultPagePriority : Pos 12, 3 Bits
+0x36c PrimaryTokenFrozen : Pos 15, 1 Bit
+0x36c ProcessVerifierTarget : Pos 16, 1 Bit
+0x36c StackRandomizationDisabled : Pos 17, 1 Bit
+0x36c AffinityPermanent : Pos 18, 1 Bit
+0x36c AffinityUpdateEnable : Pos 19, 1 Bit
+0x36c CrossSessionCreate : Pos 20, 1 Bit
+0x370 Flags            : Uint4B
+0x370 CreateReported   : Pos 0, 1 Bit
+0x370 NoDebugInherit   : Pos 1, 1 Bit
+0x370 ProcessExiting   : Pos 2, 1 Bit
+0x370 ProcessDelete    : Pos 3, 1 Bit
+0x370 Wow64SplitPages  : Pos 4, 1 Bit
+0x370 VmDeleted        : Pos 5, 1 Bit
+0x370 OutswapEnabled   : Pos 6, 1 Bit
+0x370 Outswapped       : Pos 7, 1 Bit
+0x370 ForkFailed       : Pos 8, 1 Bit
+0x370 Wow64VaSpace4Gb  : Pos 9, 1 Bit
+0x370 AddressSpaceInitialized : Pos 10, 2 Bits
+0x370 SetTimerResolution : Pos 12, 1 Bit
+0x370 BreakOnTermination : Pos 13, 1 Bit
+0x370 DeprioritizeViews : Pos 14, 1 Bit
+0x370 WriteWatch       : Pos 15, 1 Bit
+0x370 ProcessInSession : Pos 16, 1 Bit
+0x370 OverrideAddressSpace : Pos 17, 1 Bit
+0x370 HasAddressSpace  : Pos 18, 1 Bit
+0x370 LaunchPrefetched : Pos 19, 1 Bit
+0x370 InjectInpageErrors : Pos 20, 1 Bit
+0x370 VmTopDown        : Pos 21, 1 Bit
+0x370 ImageNotifyDone  : Pos 22, 1 Bit
+0x370 PdeUpdateNeeded  : Pos 23, 1 Bit
+0x370 VdmAllowed       : Pos 24, 1 Bit
+0x370 SmapAllowed      : Pos 25, 1 Bit
+0x370 ProcessInserted  : Pos 26, 1 Bit
+0x370 DefaultIoPriority : Pos 27, 3 Bits
+0x370 ProcessSelfDelete : Pos 30, 1 Bit
+0x370 SpareProcessFlags : Pos 31, 1 Bit
+0x374 ExitStatus       : Int4B
+0x378 Spare7           : Uint2B
+0x37a SubSystemMinorVersion : UChar
+0x37b SubSystemMajorVersion : UChar
+0x37a SubSystemVersion : Uint2B
+0x37c PriorityClass    : UChar
+0x380 VadRoot          : _MM_AVL_TABLE
+0x3c0 Cookie           : Uint4B
+0x3c8 AlpcContext      : _ALPC_PROCESS_CONTEXT

lkd> dt _KPROCESS
ntdll!_KPROCESS
+0x000 Header           : _DISPATCHER_HEADER
+0x018 ProfileListHead  : _LIST_ENTRY
+0x028 DirectoryTableBase : Uint8B
+0x030 Unused0          : Uint8B
+0x038 IopmOffset       : Uint2B
+0x040 ActiveProcessors : Uint8B
+0x048 KernelTime       : Uint4B
+0x04c UserTime         : Uint4B
+0x050 ReadyListHead    : _LIST_ENTRY
+0x060 SwapListEntry    : _SINGLE_LIST_ENTRY
+0x068 InstrumentationCallback : Ptr64 Void
+0x070 ThreadListHead   : _LIST_ENTRY
+0x080 ProcessLock      : Uint8B
+0x088 Affinity         : Uint8B
+0x090 AutoAlignment    : Pos 0, 1 Bit
+0x090 DisableBoost     : Pos 1, 1 Bit
+0x090 DisableQuantum   : Pos 2, 1 Bit
+0x090 ReservedFlags    : Pos 3, 29 Bits
+0x090 ProcessFlags     : Int4B
+0x094 BasePriority     : Char
+0x095 QuantumReset     : Char
+0x096 State            : UChar
+0x097 ThreadSeed       : UChar
+0x098 PowerState       : UChar
+0x099 IdealNode        : UChar
+0x09a Visited          : UChar
+0x09b Flags            : _KEXECUTE_OPTIONS
+0x09b ExecuteOptions   : UChar
+0x0a0 StackCount       : Uint8B
+0x0a8 ProcessListEntry : _LIST_ENTRY
+0x0b8 CycleTime        : Uint8B

Working set list, MMWSL (p. 340) - I guessed the structure name right:

lkd> dt _MMWSL
nt!_MMWSL
+0x000 FirstFree        : Uint4B
+0x004 FirstDynamic     : Uint4B
+0x008 LastEntry        : Uint4B
+0x00c NextSlot         : Uint4B
+0x010 Wsle             : Ptr64 _MMWSLE
+0x018 LowestPagableAddress : Ptr64 Void
+0x020 LastInitializedWsle : Uint4B
+0x024 NextEstimationSlot : Uint4B
+0x028 NextAgingSlot    : Uint4B
+0x02c EstimatedAvailable : Uint4B
+0x030 GrowthSinceLastEstimate : Uint4B
+0x034 NumberOfCommittedPageTables : Uint4B
+0x038 VadBitMapHint    : Uint4B
+0x03c NonDirectCount   : Uint4B
+0x040 LastVadBit       : Uint4B
+0x044 MaximumLastVadBit : Uint4B
+0x048 LastAllocationSizeHint : Uint4B
+0x04c LastAllocationSize : Uint4B
+0x050 NonDirectHash    : Ptr64 _MMWSLE_NONDIRECT_HASH
+0x058 HashTableStart   : Ptr64 _MMWSLE_HASH
+0x060 HighestPermittedHashAddress : Ptr64 _MMWSLE_HASH
+0x068 HighestUserAddress : Ptr64 Void
+0x070 MaximumUserPageTablePages : Uint4B
+0x074 MaximumUserPageDirectoryPages : Uint4B
+0x078 CommittedPageTables : Ptr64 Uint4B
+0x080 NumberOfCommittedPageDirectories : Uint4B
+0x088 CommittedPageDirectories : [128] Uint8B
+0x488 NumberOfCommittedPageDirectoryParents : Uint4B
+0x490 CommittedPageDirectoryParents : [1] Uint8B

PEB (pp. 341 - 342) - here’s x64 PEB structure from W2K8:

lkd> dt _PEB
ntdll!_PEB
+0x000 InheritedAddressSpace : UChar
+0x001 ReadImageFileExecOptions : UChar
+0x002 BeingDebugged    : UChar
+0x003 BitField         : UChar
+0x003 ImageUsesLargePages : Pos 0, 1 Bit
+0x003 IsProtectedProcess : Pos 1, 1 Bit
+0x003 IsLegacyProcess  : Pos 2, 1 Bit
+0x003 IsImageDynamicallyRelocated : Pos 3, 1 Bit
+0x003 SkipPatchingUser32Forwarders : Pos 4, 1 Bit
+0x003 SpareBits        : Pos 5, 3 Bits
+0x008 Mutant           : Ptr64 Void
+0x010 ImageBaseAddress : Ptr64 Void
+0x018 Ldr              : Ptr64 _PEB_LDR_DATA
+0x020 ProcessParameters : Ptr64 _RTL_USER_PROCESS_PARAMETERS
+0x028 SubSystemData    : Ptr64 Void
+0x030 ProcessHeap      : Ptr64 Void
+0x038 FastPebLock      : Ptr64 _RTL_CRITICAL_SECTION
+0x040 AtlThunkSListPtr : Ptr64 Void
+0x048 IFEOKey          : Ptr64 Void
+0x050 CrossProcessFlags : Uint4B
+0x050 ProcessInJob     : Pos 0, 1 Bit
+0x050 ProcessInitializing : Pos 1, 1 Bit
+0x050 ProcessUsingVEH  : Pos 2, 1 Bit
+0x050 ProcessUsingVCH  : Pos 3, 1 Bit
+0x050 ReservedBits0    : Pos 4, 28 Bits
+0x058 KernelCallbackTable : Ptr64 Void
+0x058 UserSharedInfoPtr : Ptr64 Void
+0x060 SystemReserved   : [1] Uint4B
+0x064 SpareUlong       : Uint4B
+0x068 SparePebPtr0     : Uint8B
+0x070 TlsExpansionCounter : Uint4B
+0x078 TlsBitmap        : Ptr64 Void
+0x080 TlsBitmapBits    : [2] Uint4B
+0x088 ReadOnlySharedMemoryBase : Ptr64 Void
+0x090 HotpatchInformation : Ptr64 Void
+0x098 ReadOnlyStaticServerData : Ptr64 Ptr64 Void
+0x0a0 AnsiCodePageData : Ptr64 Void
+0x0a8 OemCodePageData  : Ptr64 Void
+0x0b0 UnicodeCaseTableData : Ptr64 Void
+0x0b8 NumberOfProcessors : Uint4B
+0x0bc NtGlobalFlag     : Uint4B
+0x0c0 CriticalSectionTimeout : _LARGE_INTEGER
+0x0c8 HeapSegmentReserve : Uint8B
+0x0d0 HeapSegmentCommit : Uint8B
+0x0d8 HeapDeCommitTotalFreeThreshold : Uint8B
+0x0e0 HeapDeCommitFreeBlockThreshold : Uint8B
+0x0e8 NumberOfHeaps    : Uint4B
+0x0ec MaximumNumberOfHeaps : Uint4B
+0x0f0 ProcessHeaps     : Ptr64 Ptr64 Void
+0x0f8 GdiSharedHandleTable : Ptr64 Void
+0x100 ProcessStarterHelper : Ptr64 Void
+0x108 GdiDCAttributeList : Uint4B
+0x110 LoaderLock       : Ptr64 _RTL_CRITICAL_SECTION
+0x118 OSMajorVersion   : Uint4B
+0x11c OSMinorVersion   : Uint4B
+0x120 OSBuildNumber    : Uint2B
+0x122 OSCSDVersion     : Uint2B
+0x124 OSPlatformId     : Uint4B
+0x128 ImageSubsystem   : Uint4B
+0x12c ImageSubsystemMajorVersion : Uint4B
+0x130 ImageSubsystemMinorVersion : Uint4B
+0x138 ActiveProcessAffinityMask : Uint8B
+0x140 GdiHandleBuffer  : [60] Uint4B
+0x230 PostProcessInitRoutine : Ptr64     void
+0x238 TlsExpansionBitmap : Ptr64 Void
+0x240 TlsExpansionBitmapBits : [32] Uint4B
+0x2c0 SessionId        : Uint4B
+0x2c8 AppCompatFlags   : _ULARGE_INTEGER
+0x2d0 AppCompatFlagsUser : _ULARGE_INTEGER
+0x2d8 pShimData        : Ptr64 Void
+0x2e0 AppCompatInfo    : Ptr64 Void
+0x2e8 CSDVersion       : _UNICODE_STRING
+0x2f8 ActivationContextData : Ptr64 _ACTIVATION_CONTEXT_DATA
+0x300 ProcessAssemblyStorageMap : Ptr64 _ASSEMBLY_STORAGE_MAP
+0x308 SystemDefaultActivationContextData : Ptr64 _ACTIVATION_CONTEXT_DATA
+0x310 SystemAssemblyStorageMap : Ptr64 _ASSEMBLY_STORAGE_MAP
+0x318 MinimumStackCommit : Uint8B
+0x320 FlsCallback      : Ptr64 _FLS_CALLBACK_INFO
+0x328 FlsListHead      : _LIST_ENTRY
+0x338 FlsBitmap        : Ptr64 Void
+0x340 FlsBitmapBits    : [4] Uint4B
+0x350 FlsHighIndex     : Uint4B
+0x358 WerRegistrationData : Ptr64 Void
+0x360 WerShipAssertPtr : Ptr64 Void

PEB and pointers to process heap (p. 340) - couldn’t find them after PEB on x86 and x64. Needs more clarification:

7: kd> !peb
PEB at 7ffdb000
[...]

7: kd> dt _PEB
ntdll!_PEB
[...]
+0x22c FlsHighIndex     : Uint4B

7: kd> dd 7ffdb000 +0x22c +4
7ffdb230  00000000 00000000 00000000 00000000
7ffdb240  00000000 00000000 00000000 00000000
7ffdb250  00000000 00000000 00000000 00000000
7ffdb260  00000000 00000000 00000000 00000000
7ffdb270  00000000 00000000 00000000 00000000
7ffdb280  00000000 00000000 00000000 00000000
7ffdb290  00000000 00000000 00000000 00000000
7ffdb2a0  00000000 00000000 00000000 00000000

I was really annoyed by one application that I use frequently that displayed pop up message boxes reporting some problems and setting focus to itself when I was working inside other unrelated applications. Even CtxHideEx32 employed to hide message boxes didn’t help (although I should have tried to hide the main app window instead). Suddenly a blessing idea came to me to attach WinDbg to it. The GUI annoyances disappeared and now when I need that application functionality I use g command and when I don’t need it I break into it.

- Dmitry Vostokov @ DumpAnalysis.org -

Google Analytics shows the following crash dump analysis pattern frequencies to be fully analyzed later next week:

- Dmitry Vostokov @ DumpAnalysis.org -

According to Google Analytics the number of visits / year increased by 34% since 2008 with almost 150,000 unique visitors (20% increase) from 183 countries (180 in 2008) and more than 35% of them are coming back (2% increase). Here are the top 100 network locations out of 27,300 (16% increase):

Almost 75,000 Google search keywords (more than 13% increase since 2008) pointed to the portal and this blog with 100 most frequent (some are in Russian):

Special thanks to almost 1,100 web sites (16% increase) that mention the portal and this blog with the first top 100:

Top 25 visiting countries:

More than 10,000 portal and blog pages were viewed a total of almost 450,000 times with top 100 content pages:

What will you do confronted with a one million trace messages recorded between 10:44:15 and 10:46:55 with an average trace statement current of 7,000 msg/s from dozens of modules and having a one sentence problem description? One solution is to try to search for a specific vocabulary relevant to the problem description, for example, if a problem is an intermittent re-authentication then we might try to search for a word “password” or a similar one drawn from a troubleshooting domain vocabulary. So it is useful to have a Vocabulary Index to search for. Hence, the same name of this pattern. In our trace example, the search for “password” jumps straight to a small activity region of authorization modules starting from the message number #180,010 and the last “password” occurrence is in the message #180,490 that narrows initial analysis region to just 500 messages. Note the similarity here between a book and its index and a trace as a software narrative and its vocabulary index.

- Dmitry Vostokov @ TraceAnalysis.org -

Thanks to Sonny Mir who pointed to !filecache WinDbg command to diagnose low VACB (Virtual Address Control Block or View Address Control Block) conditions I was able to discern another Insufficient Memory pattern for control blocks in general. Certain system and subsystem architectures and designs may put a hard limit on the amount of data structures created to manage resources. If there is a dependency on such resources from other subsystems there could be starvation and blockage conditions resulting in a sluggish system behaviour, absence of a functional response and even in some cases a perceived system, service or application freeze.

7: kd> !filecache
***** Dump file cache******
  Reading and sorting VACBs ...
  Removed 0 nonactive VACBs, processing 1907 active VACBs …
File Cache Information
  Current size 408276 kb
  Peak size    468992 kb
  1907 Control Areas
[…]

I plan to add more insufficient control block case studies including user space.

- Dmitry Vostokov @ DumpAnalysis.org -

Main topics of Debugged! MZ/PE magazine issues for June and September 2010:

Debugged! MZ/PE: Multiplatform Software Defects, June, 2010 (Paperback, ISBN: 978-1906717902)

Debugged! MZ/PE: Software Defect Visualization and Annotation, September, 2010 (Paperback, ISBN: 978-1906717919)

Here are draft front covers designed today:

- Dmitry Vostokov @ DumpAnalysis.org -

See the greeting card on the portal together with New Year’s Eve code analysis puzzle:

DumpAnalysis.org Wishes Happy New Year 7DA!

- Dmitry Vostokov @ DumpAnalysis.org -

Memorianity soon to publish its Testament with 7 microkernel prophecies, childhood universal memory dump visions of its founder, the recollection of a conversion and other supporting materials. This full color scripture is small to carry around:

Title: Memory Religion: A Testament
ISBN-13: 978-1906717476
Pages: 24

The cover image is an allegorical interpretation of the concept of the Original Defect:

- Dmitry Vostokov @ Memory Religion Portal -

On the next day in Townley Hall library, after submitting Volume 3 of Memory Dump Analysis to print:

- Dmitry Vostokov @ DumpAnalysis.org -

This is a book I bought a few years ago and started reading immediately but put aside and only this summer read it fully from cover to cover. In order to appreciate its content you need some degree of mathematical and computer science maturity. For example, if you have never heard of his theorems and only read Incompleteness: The Proof and Paradox of Kurt Godel or similar popular book then you would have difficulty going through the book and it would appear boring. It is not an entertaining or bedside reading. This is why I put it aside on the first reading although I knew about this theorem since I read “Mathematics: The Loss of Certainty” more than 25 years ago being a schoolboy (in Russian translation). Just before writing this review I ordered “There’s Something About Godel: The Complete Guide to the Incompleteness Theorem” and the latter looks like less heavy reading judged from excerpts from its publisher website. Putting all these reminiscences aside I really enjoyed second reading of “Godel’s Theorem”. It really clarified some points from ¬B->¬A or PA & ¬Con(PA) perspectives and made me curious about fixpoints. I even borrowed the latter term and introduced them for crash dump analysis and debugging: “a dereference fixpoint”. I also liked chapters 4 and 6 about using Godel’s theorems outside mathematics and clarifying misconceptions in Rucker’s and Penrose’s books. However, after a few months I cannot recall anything definite what I read from that book although I felt good that I understood everything while reading so perhaps the book requires the 3rd reading for me I’m going to give it another try after “There’s Something About Godel” and update this review.

Godel’s Theorem: An Incomplete Guide to Its Use and Abuse

- Dmitry Vostokov @ LiterateScientist.com -

When looking at crash dumps it is good to keep an eye on new API that might surface on stack traces and in component relationships. Plan to order this book tomorrow and put my reading notes on Software Generalist blog:

Introducing Windows® 7 for Developers

- Dmitry Vostokov @ DumpAnalysis.org -

a mad day - a day spent doing memory (dump) analysis and/or debugging

Examples: What a mad day! We had several blokes at a bobo address. Those events were rather sad.

- Dmitry Vostokov @ DumpAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

I read this book in just one day from cover to cover. I’m not a professional biologist and learnt about evolution 25 - 30 years ago from Marxist perspective. My understanding of evolution has greatly improved this year after reading Darwin’s Dangerous Idea, This Is Biology, Breaking the Spell, Evolution: The First Four Billion Years and The 10,000 Year Explosion books. I’ve also started reading (and listening to its unabridged version on CDs simultaneously) the latest Dawkins’ book “The Greatest Show on Earth” (to be reviewed as soon as I finish) after the thought “Who’s that guy?” finally tipped. I noticed the partnership of D. Dennett and R. Dawkins when reading books and also rants from religious camps when reading reviews. So I was very keen to read the promised history of Dawkins thought in “The Selfish Genius” book and I really enjoyed it. Judged from the background knowledge I acquired while reading various books about evolution “The Selfish Genius” seems fair and balanced. Sometimes it reminded me the similar problem in Physics: String Theory vs. Others (Not Even Wrong and the Trouble With Physics). When I put “The Selfish Genius” and resumed reading “The Greatest Show on Earth” I immediately noticed a footnote on page 216 (ISBN 978-1-4165-9478-9): “epigenetics, a modish buzz-word now enjoying its fifteen minutes” and if you are curious about the source of this anger read “The Selfish Genius” book. I also like the point of the book that for different people with different backgrounds “Evolution” means different things. For me it is about evolution of software but mainly about evolution of software defects: Darwinian Debugging and I even bugtated Dawkins’ meme: Bugtation No.108.

The Selfish Genius: How Richard Dawkins Rewrote Darwin’s Legacy

- Dmitry Vostokov @ LiterateScientist.com -

I propose to celebrate it on 08.08 every year starting from The Year of Dump Analysis, 2010, 7DA at 8:00 (I prefer 8pm for MAD Day, moderation is important in debugging too).

What do you think? If you count things from 0 or favour user space there is an alternative date: 07.07

- Dmitry Vostokov @ DumpAnalysis.org -

A kernel dump from a frozen system shows an executive resource wait chain:

0: kd> !locks
[...] 
Resource @ driverA!Resource (0xf58de4e0)    Exclusively owned
    Contention Count = 4411
    NumberOfExclusiveWaiters = 11
     Threads: 86d14ae8-01<*>
     Threads Waiting On Exclusive Access:
              8a788db0       8750e970       86c568a0       897ed428      
              86e34db0       86ca8ac0       86b22020       86fef5d8      
              872abdb0       86d16750       87b55830      
[…]

The blocking thread 86d14ae8 had been blocked waiting for a notification event for more than 2 hours:

0: kd> !thread 86d14ae8 1f
THREAD 86d14ae8  Cid 0004.29c4  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
    b81e7adc  NotificationEvent
Not impersonating
DeviceMap                 e1001830
Owning Process            8a78b020       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      8378144        Ticks: 503606 (0:02:11:08.843)
Context Switch Count      1016            
UserTime                  00:00:00.000
KernelTime                00:00:00.015
Start Address driverA!WorkerThreadDispatcher (0xf596ea0e)
Stack Init b81e8000 Current b81e7a2c Base b81e8000 Limit b81e5000 Call 0
Priority 14 BasePriority 10 PriorityDecrement 4
ChildEBP RetAddr 
b81e7a44 8083d5b1 nt!KiSwapContext+0×26
b81e7a70 8083df9e nt!KiSwapThread+0×2e5
b81e7ab8 f59d374d nt!KeWaitForSingleObject+0×346
[…]
b81e7b48 f59b9289 driverB!TcpDisconnect+0×42
[…]
b81e7c40 f595a8a5 nt!IofCallDriver+0×45
b81e7c48 f595ba1e driverA!SubmitTdiRequestNoWait+0×28
[…]
b81e7dac 80920833 driverA!WorkerThreadDispatcher+0×1a
b81e7ddc 8083fe9f nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

We see that the wait happens after requesting a TCP disconnect so we check the list of IRP to see if there is any distribution anomaly among pending IRP:

0: kd> !irpfind
  Irp    [ Thread ] irpStack: (Mj,Mn)   DevObj  [Driver]         MDL Process
[...]
86c68d98 [88d2bdb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c6a5c0 [89b118c0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c6b008 [87564b40] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c6caf0 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c7bb28 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c7bd98 [8753ddb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c80008 [88d7b378] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c80590 [88e1c368] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c845a8 [89d2b400] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c84b80 [88d7b378] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c86008 [88e1c368] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c86688 [86d9a788] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c86d98 [88d2bdb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c87990 [88e1c368] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c8b640 [8757c3f0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c8f368 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c8f650 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c92590 [87625c30] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c92bc8 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c94008 [8757c3f0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c94318 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c9a308 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c9e008 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c9e308 [89d2b400] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0350 [87638020] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0870 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0b28 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0d98 [86db0db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca4918 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca6878 [87564b40] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86caa458 [88d7b378] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cacc20 [86d4fb40] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cb0818 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cb3658 [87638020] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cb9d98 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
[…]

Indeed, we see a high disproportion of TCP I/O requests (many hundreds) after exporting command output to Excel:

We check all stack traces and see one system thread trying to clean TCP connection blocked for almost the same time (more than 2 hours):

0: kd> !stacks
Proc.Thread  .Thread  Ticks   ThreadState Blocker
                            [8a78b020 System]
[...]
   4.00268c  870cf768 00765bd Blocked    tcpip!TCPCleanup+0xcf
[…]

0: kd> !whattime  00765bd
484797 Ticks in Standard Time:   2:06:14.953s

0: kd> !thread 870cf768 1f
THREAD 870cf768  Cid 0004.268c  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
    870a01f4  SynchronizationEvent
IRP List:
    8726fb00: (0006,0268) Flags: 00000404  Mdl: 00000000
Not impersonating
DeviceMap                 e1001830
Owning Process            8a78b020       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      8396953        Ticks: 484797 (0:02:06:14.953)
Context Switch Count      537            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address nt!ExpWorkerThread (0×8082da4b)
Stack Init b87b0000 Current b87afa18 Base b87b0000 Limit b87ad000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0
ChildEBP RetAddr 
b87afa30 8083d5b1 nt!KiSwapContext+0×26
b87afa5c 8083df9e nt!KiSwapThread+0×2e5
b87afaa4 f5a9f9a6 nt!KeWaitForSingleObject+0×346
b87afaf0 f5a96a9d tcpip!TCPCleanup+0xcf
b87afb2c 80840153 tcpip!TCPDispatch+0×10c
b87afb40 f75eb817 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
b87afb64 f75e8698 driverC!DispatchPassThrough+0×4c
[…]
b87afbcc 8092ec0a nt!IofCallDriver+0×45
b87afbfc 8092b6af nt!IopCloseFile+0×2ae
b87afc2c 8092b852 nt!ObpDecrementHandleCount+0xcc
b87afc54 8092b776 nt!ObpCloseHandleTableEntry+0×131
b87afc98 8092b7c1 nt!ObpCloseHandle+0×82
b87afca8 80833bdf nt!NtClose+0×1b
b87afca8 8083b00c nt!KiFastCallEntry+0xfc (TrapFrame @ b87afcb4)
b87afd24 f59d3a3a nt!ZwClose+0×11
b87afd3c f59b78a1 driverB!TdiCloseConnection+0×38
[…]
b87afdac 80920833 nt!ExpWorkerThread+0xeb
b87afddc 8083fe9f nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

- Dmitry Vostokov @ DumpAnalysis.org -

Completely mutated Richard Dawkins‘ quotation after listening to his latest book “The Greatest Show on Earth” and reading Fern Elsdon-Baker’s book “The Selfish Genius”:

“Memory dumps are facts”.

Dmitry Vostokov’s statement upon hearing “Evolution is a fact.”

No offence to Dawkins’ camp, watch out the publication of the next issue of Debugged! magazine about systematics and evolution of software defects:

Darwinian Debugging

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday.

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

Finally, after the long delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Software Tracing

- Dmitry Vostokov @ DumpAnalysis.org -