Software Diagnostics Library

Befind every trace and its messages is source code:

Borrowing the acronym PLOT (Program Lines of Trace) we now try to discern basic source code patterns that give rise to simple message patterns in software traces. There are only a few distinct PLOTs and the ability to mentally map trace statements to source code is crucial to software trace reading and comprehension. More about that in subsequent parts. More complex message patterns (for example, specific message blocks or correlated messages) arise from supportable and maintainable realizations of architectural, design and implementation patterns and will be covered in another post series.

I was thinking about acronym SLOT (Source Lines of Trace) but decided to use PLOT because it metaphorically bijects into literary theory and narrative plots.

Forthcoming CDF and ETW Software Trace Analysis: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

PLOT - Program Lines of Trace - the source code lines behind trace messages

Examples: What a plot do we have here! The struggle against the monster database component and endless voyages across space boundaries.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

Today we introduce an icon for Stack Overflow (user mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Stack Overflow (kernel mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Modern pattern-driven software trace analysis on Microsoft and Citrix platforms urgently requires a practical guide and OpenTask plans to publish this summer the following book in both Practical Foundations and Systematic Software Fault Analysis series:

• Title: Citrix Common Diagnostic Facility (CDF) and Microsoft Event Tracing for Windows (ETW) Software Trace Analysis: Practical Foundations
• Author: Dmitry Vostokov
• Publisher: Opentask (August 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN: 1906717176
• ISBN-13: 978-1906717179
• Paperback: 200 pages

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

c’t – Magazin für Computertechnik has published a review of First Fault Software Problem Solving book: 

http://www.heise.de/ct/inhalt/2010/08/192/ (in German)

Fabian Röken kindly translated it into English:

No single large software package comes without errors. It seems that customers simply accept this, patiently waiting and hoping for patches or updates. Skwire sticks up for a more target-aimed approach: one will never get a faultless software, but it would already be a great improvement if flaws were already solved on their first occurrence (”first fault”) and not only after a long analysis (”second fault”).

The advantages are actually obvious. However, a corresponding stringent system architecture, as common on mainframes such as IBM’s z/OS, did not become prevalent in the PC market.

Skwire outlines the types of errors and strategies to resolve them in all details. His 40 years of experience, such as at IBM, shimmers through again and again. He puts emphasis on making sure that the reader understands the terminology he is using: “What is a problem in the first place?”, “What is a service point?” - in some cases he also explains specific metrics such as the “serviceability rating”.

His tool classification includes teaching tips, e.g. regarding the structure of a protocol in case of errors; or for tracking the important information how often an error must occur before a solution has to be approached. His suggestions equally address developers, designers, testers, managers - and the end user. In his last chapter he presents and reviews commercial tools in the first fault and second fault environment.

Skwire addresses a topic which is unfortunately very much neglected, and this alone already makes it worth enough to take a look at his book (***). Short quotations and humorous drawings relax the technical topic. If you are looking for an overview then you will be fine with this book. However, if you are a software developer looking for source code samples then you will search in vain. Skwire has released the book under the print-on-demand process. You will find it on Amazon, for example.

(Tobias Engler/fm)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It is time to start being systematic. In addition to all-encompassing multi-volume Memory Dump Analysis Anthology OpenTask starts Systematic Software Fault Analysis series with Crash Dump Analysis: Practical Foundations as the first book. It introduces basic definitions, tools, memory dump collection and preliminary analysis methods for Windows platforms including legacy versions. This practical reference guide is a must have for system administrators of Windows server platforms and client workstations, technical support engineers and general Windows users. It builds foundation for the second book Crash Dump Analysis for System Administrators and Support Engineers and the remaining tetralogy books Windows Crash Dump Analysis and Advanced Windows Crash Dump Analysis.

Product information:

• Title: Crash Dump Analysis: Practical Foundations (Windows Edition, Systematic Software Fault Analysis Series)
• Authors: Dmitry Vostokov
• Publisher: Opentask (May 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-98-8
• Paperback: 100 pages

Fromt cover:

Table of Contents to be published soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The book is about to be submitted for publication. It has 68 full color illustrations. Here is TOC:

Table of Contents

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Module Variety pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Anthology lacks full color inserts. This is rather a current limitation of POD technology that OpenTask publisher uses at the present time. The solution previously announced a year ago is to print a separate full color title with selected articles and illustrations. Finally, it is about to be published next month. The book front and back covers are collages from covers of individual memory dump analysis volumes:

TOC will be available soon. Here is the book data:

• Title: Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
• Author: Dmitry Vostokov
• Publisher: OpenTask (May 2010)
• Language: English
• Product Dimensions: 21.6 x 14.0
• Paperback: 110 pages
• ISBN-13: 978-1906717698

OpenTask also plans a separate color supplement for volumes 4 and 5 once they are published. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Yesterday I discovered the blog j00ru//vx where I was pleased to see another memory visualization approach which I classify as synthetic:

x86 Kernel Memory Space Visualization (KernelMAP v0.0.1)

So far now I put a bit more extended (but in no way complete) classification with links (based on my previous blog post where every category is presented in chronological order of my encounter with links):

1. Synthetic

• Numerous debuggers that show memory maps for easy navigation (links to follow)
• VMMap
• KernelMAP v0.0.1
• MemSpyy
• Virtual memory usage and GC heap usage

2. Natural

a. Static

• Dump2Picture (Windows)
• dump2pic (Linux)
• 2D and 3D visualization using general-purpose tools like ParaView.

b. Semi-dynamic

• WinDbg scripts

c. Dynamic 

• Haywire

Please let me know any other approaches or links you know. 

PS. I’m currently a big fan of artificial evolution and recommend this fantastic full-color book that has good ideas about expression-based visualization:

The Art of Artificial Evolution: A Handbook on Evolutionary Art and Music (Natural Computing Series)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Spiking Thread pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A print spooler service process was hanging and blocking print-related requests from other coupled processes. Default analysis of its dump doesn’t show any problem (it shows normal service main thread):

0:000> !analyze -v

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT

STACK_TEXT: 
0006fbcc 7c82776b 77e418b2 00000064 00000000 ntdll!KiFastSystemCallRet
0006fbd0 77e418b2 00000064 00000000 00000000 ntdll!NtReadFile+0xc
0006fc38 77f65edb 00000064 0006fd04 0000021a kernel32!ReadFile+0x16c
0006fc64 77f65f82 00000064 0006fd04 0000021a advapi32!ScGetPipeInput+0x2a
0006fcd8 77f51ed9 00000064 0006fd04 0000021a advapi32!ScDispatcherLoop+0x51
0006ff3c 01004019 0100d5bc 010047a2 00000001 advapi32!StartServiceCtrlDispatcherW+0xe3
0006ff44 010047a2 00000001 00263fa0 00262be0 spoolsv!main+0xb
0006ffc0 77e6f23b 00000000 00000000 7ffd7000 spoolsv!mainCRTStartup+0x12f
0006fff0 00000000 0100468c 00000000 78746341 kernel32!BaseProcessStart+0x23

0:000> !analyze -v -hang

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

BUGCHECK_STR:  HANG

STACK_TEXT: 
0006fbcc 7c82776b 77e418b2 00000064 00000000 ntdll!KiFastSystemCallRet
0006fbd0 77e418b2 00000064 00000000 00000000 ntdll!NtReadFile+0xc
0006fc38 77f65edb 00000064 0006fd04 0000021a kernel32!ReadFile+0x16c
0006fc64 77f65f82 00000064 0006fd04 0000021a advapi32!ScGetPipeInput+0x2a
0006fcd8 77f51ed9 00000064 0006fd04 0000021a advapi32!ScDispatcherLoop+0x51
0006ff3c 01004019 0100d5bc 010047a2 00000001 advapi32!StartServiceCtrlDispatcherW+0xe3
0006ff44 010047a2 00000001 00263fa0 00262be0 spoolsv!main+0xb
0006ffc0 77e6f23b 00000000 00000000 7ffd7000 spoolsv!mainCRTStartup+0x12f
0006fff0 00000000 0100468c 00000000 78746341 kernel32!BaseProcessStart+0x23

Stack trace collection shows several threads waiting for a critical section when allocating heap blocks or calling loader functions, for example:

0:000> ~*k

[...]

  20  Id: 540.71d0 Suspend: 1 Teb: 7ffa2000 Unfrozen
ChildEBP RetAddr 
0597fa20 7c827d0b ntdll!KiFastSystemCallRet
0597fa24 7c83d236 ntdll!NtWaitForSingleObject+0xc
0597fa60 7c83d281 ntdll!RtlpWaitOnCriticalSection+0×1a3
0597fa80 7c82a264 ntdll!RtlEnterCriticalSection+0xa8
0597fca8 77c7e5cf ntdll!RtlAllocateHeap+0×313
0597fcbc 77c7e5a6 rpcrt4!AllocWrapper+0×1e
0597fcc8 77c82069 rpcrt4!operator new+0xd
0597fdf4 77c812a5 rpcrt4!LRPC_SCALL::LrpcMessageToRpcMessage+0xd0
0597fe20 77c88678 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×66
0597ff84 77c88792 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0597ff8c 77c8872d rpcrt4!RecvLotsaCallsWrapper+0xd
0597ffac 77c7b110 rpcrt4!BaseCachedThreadRoutine+0×9d
0597ffb8 77e64829 rpcrt4!ThreadStartRoutine+0×1b
0597ffec 00000000 kernel32!BaseThreadStart+0×34

  21  Id: 540.5b3c Suspend: 1 Teb: 7ff9f000 Unfrozen
ChildEBP RetAddr 
090dfea0 7c827d0b ntdll!KiFastSystemCallRet
090dfea4 7c83d236 ntdll!NtWaitForSingleObject+0xc
090dfee0 7c83d281 ntdll!RtlpWaitOnCriticalSection+0×1a3
090dff00 7c81909b ntdll!RtlEnterCriticalSection+0xa8
090dffa8 77e4f920 ntdll!LdrShutdownThread+0×33
090dffb8 77e6482e kernel32!ExitThread+0×2f
090dffec 00000000 kernel32!BaseThreadStart+0×39

[...]

!cs command shows wait chains and signs of critical section corruption. Here is the commented output:

0:000> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x7c8877c0
Critical section   = 0×7c8877a0 (ntdll!LdrpLoaderLock+0×0)
LOCKED
LockCount          = 0×5
WaiterWoken        = No
OwningThread       = 0×00005a20
RecursionCount     = 0×1
LockSemaphore      = 0×184
SpinCount          = 0×00000000
OwningThread DbgId = ~25s
OwningThread Stack =
 ChildEBP RetAddr  Args to Child            
 0568f42c 7c827d0b 7c83d236 00000da0 00000000 ntdll!KiFastSystemCallRet
 0568f430 7c83d236 00000da0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
 0568f46c 7c83d281 00000da0 00000004 00080000 ntdll!RtlpWaitOnCriticalSection+0×1a3
 0568f48c 7c82a264 00080608 7c82e6b4 0000008e ntdll!RtlEnterCriticalSection+0xa8
 0568f6b4 77e6427d 00080000 00000000 00000594 ntdll!RtlAllocateHeap+0×313
 0568f718 77e643a2 77e643d0 00020abc 00000000 kernel32!BasepComputeProcessPath+0xc2
 0568f758 77e65348 00000000 00000000 00000000 kernel32!BaseComputeProcessDllPath+0xe3
 0568f79c 77e6528f 0568f7b8 00000000 4dc5822c kernel32!GetModuleHandleForUnicodeString+0×2b
 0568fc14 77e65155 00000001 00000002 0568fc38 kernel32!BasepGetModuleHandleExW+0×17f
 0568fc2c 4dc4d554 0568fc38 003a0043 0057005c kernel32!GetModuleHandleW+0×29
 0568fe4c 4dc49a0a 4dc32328 00000001 0568fe80 MSCTFIME!GetSystemModuleHandleW+0×40
 0568fe5c 4dc49bc3 4dc5822c 4dc32328 4dc32380 MSCTFIME!GetFn+0×2e
 0568fe74 4dc49039 00000003 0568fea0 4dc49fbb MSCTFIME!TF_DllDetachInOther+0×2a
 0568fe80 4dc49fbb 4dc30000 00000003 00000000 MSCTFIME!DllMain+0×1d
 0568fea0 7c81a352 4dc30000 00000003 00000000 MSCTFIME!_DllMainCRTStartup+0×52
 0568fec0 7c819178 4dc49f69 4dc30000 00000003 ntdll!LdrpCallInitRoutine+0×14
 0568ff74 77e4f920 3533e0ec 00000000 0568ff98 ntdll!LdrShutdownThread+0xd2
 0568ff84 77e52868 00000000 3533e0ec 77e5bf51 kernel32!ExitThread+0×2f
 0568ff98 3530cd31 35100000 00000000 00000000 kernel32!FreeLibraryAndExitThread+0×40
WARNING: Stack unwind information not available. Following frames may be wrong.
 0568ffb8 77e64829 00001430 00000000 00000000 PrintDriverA!DllGetClassObject+0×1dcdb1
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.

The thread #25 is blocked waiting for the critical section 00080608 but it also owns another critical section LdrpLoaderLock and blocks 5 other threads. Stack trace features PrintDriverA module.

-----------------------------------------
DebugInfo          = 0x7c887be0
Critical section   = 0×7c887740 (ntdll!FastPebLock+0×0)
LOCKED
LockCount          = 0×0
WaiterWoken        = No
OwningThread       = 0×00005a20
RecursionCount     = 0×1
LockSemaphore      = 0×868
SpinCount          = 0×00000000
OwningThread DbgId = ~25s
OwningThread Stack =
 ChildEBP RetAddr  Args to Child            
 0568f42c 7c827d0b 7c83d236 00000da0 00000000 ntdll!KiFastSystemCallRet
 0568f430 7c83d236 00000da0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
 0568f46c 7c83d281 00000da0 00000004 00080000 ntdll!RtlpWaitOnCriticalSection+0×1a3
 0568f48c 7c82a264 00080608 7c82e6b4 0000008e ntdll!RtlEnterCriticalSection+0xa8
 0568f6b4 77e6427d 00080000 00000000 00000594 ntdll!RtlAllocateHeap+0×313
 0568f718 77e643a2 77e643d0 00020abc 00000000 kernel32!BasepComputeProcessPath+0xc2
 0568f758 77e65348 00000000 00000000 00000000 kernel32!BaseComputeProcessDllPath+0xe3
 0568f79c 77e6528f 0568f7b8 00000000 4dc5822c kernel32!GetModuleHandleForUnicodeString+0×2b
 0568fc14 77e65155 00000001 00000002 0568fc38 kernel32!BasepGetModuleHandleExW+0×17f
 0568fc2c 4dc4d554 0568fc38 003a0043 0057005c kernel32!GetModuleHandleW+0×29
 0568fe4c 4dc49a0a 4dc32328 00000001 0568fe80 MSCTFIME!GetSystemModuleHandleW+0×40
 0568fe5c 4dc49bc3 4dc5822c 4dc32328 4dc32380 MSCTFIME!GetFn+0×2e
 0568fe74 4dc49039 00000003 0568fea0 4dc49fbb MSCTFIME!TF_DllDetachInOther+0×2a
 0568fe80 4dc49fbb 4dc30000 00000003 00000000 MSCTFIME!DllMain+0×1d
 0568fea0 7c81a352 4dc30000 00000003 00000000 MSCTFIME!_DllMainCRTStartup+0×52
 0568fec0 7c819178 4dc49f69 4dc30000 00000003 ntdll!LdrpCallInitRoutine+0×14
 0568ff74 77e4f920 3533e0ec 00000000 0568ff98 ntdll!LdrShutdownThread+0xd2
 0568ff84 77e52868 00000000 3533e0ec 77e5bf51 kernel32!ExitThread+0×2f
 0568ff98 3530cd31 35100000 00000000 00000000 kernel32!FreeLibraryAndExitThread+0×40
WARNING: Stack unwind information not available. Following frames may be wrong.
 0568ffb8 77e64829 00001430 00000000 00000000 PrintDriverA!DllGetClassObject+0×1dcdb1
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.

This is the same thread #25 but it also owns another critical section FastPebLock but this doesn’t block additional threads.

-----------------------------------------
DebugInfo          = 0x7c887c80
Critical section   = 0×00080608 (+0×80608)
LOCKED
LockCount          = 0×4
WaiterWoken        = No
OwningThread       = 0×0000a8c4
RecursionCount     = 0×1
LockSemaphore      = 0xDA0
SpinCount          = 0×00000fa0
OwningThread DbgId = ~22s
OwningThread Stack =
 ChildEBP RetAddr  Args to Child            
 03456830 7739bf53 7739610a 00000000 00000000 ntdll!KiFastSystemCallRet
 03456868 7738965e 186403ba 00000000 00000001 user32!NtUserWaitMessage+0xc
 03456890 7739f762 77380000 05bdc880 00000000 user32!InternalDialogBox+0xd0
 03456b50 7739f047 03456cac 00000000 ffffffff user32!SoftModalMessageBox+0×94b
 03456ca0 7739eec9 03456cac 00000028 00000000 user32!MessageBoxWorker+0×2ba
 03456cf8 773d7d0d 00000000 0ae7cc20 02639ea8 user32!MessageBoxTimeoutW+0×7a
 03456d80 773c42c8 00000000 03456e14 03456df4 user32!MessageBoxTimeoutA+0×9c
 03456da0 773c42a4 00000000 03456e14 03456df4 user32!MessageBoxExA+0×1b
 03456dbc 6dfcf8c2 00000000 03456e14 03456df4 user32!MessageBoxA+0×45
 034575f8 6dfd05cf 03456e5a 03457624 77bc6cd5 compstui!FilterException+0×174
 03458584 6dfcff1e 02638dc8 00000000 03458c58 compstui!CallpfnPSUI+0×110
 034587f0 6dfd00a2 02638b40 026393f8 00000000 compstui!InsertPSUIPage+0×201
 03458848 7307c9ae 43440001 00000005 02118690 compstui!CPSUICallBack+0xed
 03458870 6dfd059a 0345888c 03458c58 7307c8da winspool!DevicePropertySheets+0xd4
 034588d4 6dfcff1e 026393f8 00000000 03458c58 compstui!CallpfnPSUI+0xdb
 03458b40 6dfd00a2 02638b40 02638b40 00000000 compstui!InsertPSUIPage+0×201
 03458b98 6dfd06a3 43440000 00000005 7307c8da compstui!CPSUICallBack+0xed
 03458bcc 6dfd0799 00000000 7307c8da 03458c58 compstui!DoCommonPropertySheetUI+0×74
 03458be4 730801c5 00000000 7307c8da 03458c58 compstui!CommonPropertySheetUIW+0×17
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.

The thread #22 is blocked waiting for the message box but it also owns the critical section 00080608 we have seen above and the thread blocks 4 other threads.

Cannot read structure field value at 0x04ddbb64, error 0
Cannot determine if the critical section is locked or not.
-----------------------------------------
Critical section   = 0x04ddbb60 (+0x4DDBB60)
DebugInfo          = 0x01e10858
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
-----------------------------------------
DebugInfo          = 0x05b24d38
Critical section   = 0x0589de08 (PrintDriverB+0×49DE8)
LOCKED
LockCount          = 0xC5D3FFF
WaiterWoken        = Yes
OwningThread       = 0×00008487
RecursionCount     = 0×8DD5FF50
LockSemaphore      = 0×50CE8B00
SpinCount          = 0×878dd5ff

WARNING: critical section DebugInfo = 0x0080878d doesn't point back
to the DebugInfo found in the active critical sections list = 0x05b24d38.
The critical section was probably reused without calling DeleteCriticalSection.

ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
-----------------------------------------
DebugInfo          = 0x05b24c98
Critical section   = 0x0589ddd8 (PrintDriverB+0x49DB8)
LOCKED
LockCount          = 0x1D38F6EE
WaiterWoken        = Yes
OwningThread       = 0x1c2444db
RecursionCount     = 0xD3FF50CE
LockSemaphore      = 0x8D04EC83
SpinCount          = 0x1cd9744f

WARNING: critical section DebugInfo = 0x8dffff73 doesn't point back
to the DebugInfo found in the active critical sections list = 0x05b24c98.
The critical section was probably reused without calling DeleteCriticalSection.

Cannot read structure field value at 0x8dffff75, error 0
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
-----------------------------------------
DebugInfo          = 0x05b24f40
Critical section   = 0x0589de28 (PrintDriverB+0x49E08)
LOCKED
LockCount          = 0x1D38F6EE
WaiterWoken        = Yes
OwningThread       = 0x1c2444db
RecursionCount     = 0xD3FF50CE
LockSemaphore      = 0x8D04EC83
SpinCount          = 0x00008c8f

WARNING: critical section DebugInfo = 0x8d242454 doesn't point back
to the DebugInfo found in the active critical sections list = 0x05b24f40.
The critical section was probably reused without calling DeleteCriticalSection.

Cannot read structure field value at 0x8d242456, error 0
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
-----------------------------------------
DebugInfo          = 0x05b24d10
Critical section   = 0x0589de08 (PrintDriverB+0x49DE8)
LOCKED
LockCount          = 0xC5D3FFF
WaiterWoken        = Yes
OwningThread       = 0x00008487
RecursionCount     = 0x8DD5FF50
LockSemaphore      = 0x50CE8B00
SpinCount          = 0x878dd5ff

WARNING: critical section DebugInfo = 0x0080878d doesn't point back
to the DebugInfo found in the active critical sections list = 0x05b24d10.
The critical section was probably reused without calling DeleteCriticalSection.

ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
-----------------------------------------
DebugInfo          = 0x05b24ec8
Critical section   = 0x0589de28 (PrintDriverB+0x49E08)
LOCKED
LockCount          = 0x1D38F6EE
WaiterWoken        = Yes
OwningThread       = 0x1c2444db
RecursionCount     = 0xD3FF50CE
LockSemaphore      = 0x8D04EC83
SpinCount          = 0x00008c8f

WARNING: critical section DebugInfo = 0x8d242454 doesn't point back
to the DebugInfo found in the active critical sections list = 0x05b24ec8.
The critical section was probably reused without calling DeleteCriticalSection.

Cannot read structure field value at 0x8d242456, error 0
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
-----------------------------------------
DebugInfo          = 0x05b24cc0
Critical section   = 0x0589ddd8 (PrintDriverB+0x49DB8)
LOCKED
LockCount          = 0x1D38F6EE
WaiterWoken        = Yes
OwningThread       = 0x1c2444db
RecursionCount     = 0xD3FF50CE
LockSemaphore      = 0x8D04EC83
SpinCount          = 0x1cd9744f

WARNING: critical section DebugInfo = 0x8dffff73 doesn't point back
to the DebugInfo found in the active critical sections list = 0x05b24cc0.
The critical section was probably reused without calling DeleteCriticalSection.

Cannot read structure field value at 0x8dffff75, error 0
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.

Here we see the recurrence of PrintDriverB module in the output that looks like corruption. Because the thread #22 heads the wait chain we look at its full stack trace:

0:000> ~22s; kL 100
eax=00465758 ebx=00000000 ecx=00467514 edx=00000001 esi=00467500 edi=00000000
eip=7c8285ec esp=03456834 ebp=03456868 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
7c8285ec c3              ret
ChildEBP RetAddr
03456830 7739bf53 ntdll!KiFastSystemCallRet
03456868 7738965e user32!NtUserWaitMessage+0xc
03456890 7739f762 user32!InternalDialogBox+0xd0
03456b50 7739f047 user32!SoftModalMessageBox+0x94b
03456ca0 7739eec9 user32!MessageBoxWorker+0x2ba
03456cf8 773d7d0d user32!MessageBoxTimeoutW+0x7a
03456d80 773c42c8 user32!MessageBoxTimeoutA+0x9c
03456da0 773c42a4 user32!MessageBoxExA+0x1b
03456dbc 6dfcf8c2 user32!MessageBoxA+0×45
034575f8 6dfd05cf compstui!FilterException+0×174
03458584 6dfcff1e compstui!CallpfnPSUI+0×110
034587f0 6dfd00a2 compstui!InsertPSUIPage+0×201
03458848 7307c9ae compstui!CPSUICallBack+0xed
03458870 6dfd059a winspool!DevicePropertySheets+0xd4
034588d4 6dfcff1e compstui!CallpfnPSUI+0xdb
03458b40 6dfd00a2 compstui!InsertPSUIPage+0×201
03458b98 6dfd06a3 compstui!CPSUICallBack+0xed
03458bcc 6dfd0799 compstui!DoCommonPropertySheetUI+0×74
03458be4 730801c5 compstui!CommonPropertySheetUIW+0×17
03458c2c 73080f5d winspool!CallCommonPropertySheetUI+0×43
03459074 35145947 winspool!PrinterPropertiesNative+0×10c
034590c4 3513a045 PrintDriverA!DllGetClassObject+0×159c7
0345e9ac 35131819 PrintDriverA!DllGetClassObject+0xa0c5
0345ebdc 32020661 PrintDriverA!DllGetClassObject+0×1899
0345ec04 3201b171 PS5UI!HComOEMPrinterEvent+0×33
0345ec48 02117a79 PS5UI!DrvPrinterEvent+0×239
0345eea4 7308218c PrintDriverA!DrvPrinterEvent+0xf9
0345eef0 761542cc winspool!SpoolerPrinterEventNative+0×57
0345ef0c 76155fd6 localspl!SplDriverEvent+0×21
0345ef30 76144799 localspl!PrinterDriverEvent+0×46
0345f3f8 76144ab2 localspl!SplAddPrinter+0×5f3
0345f424 74070193 localspl!LocalAddPrinterEx+0×2e
0345f874 7407025c spoolss!AddPrinterExW+0×151
0345f890 0100792d spoolss!AddPrinterW+0×17
0345f8ac 01006762 spoolsv!YAddPrinter+0×75
0345f8d0 77c80193 spoolsv!RpcAddPrinter+0×37
0345f8f8 77ce33e1 rpcrt4!Invoke+0×30
0345fcf8 77ce35c4 rpcrt4!NdrStubCall2+0×299
0345fd14 77c7ff7a rpcrt4!NdrServerCall2+0×19
0345fd48 77c8042d rpcrt4!DispatchToStubInCNoAvrf+0×38
0345fd9c 77c80353 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
0345fdc0 77c811dc rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
0345fdfc 77c812f0 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×42c
0345fe20 77c88678 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0345ff84 77c88792 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0345ff8c 77c8872d rpcrt4!RecvLotsaCallsWrapper+0xd
0345ffac 77c7b110 rpcrt4!BaseCachedThreadRoutine+0×9d
0345ffb8 77e64829 rpcrt4!ThreadStartRoutine+0×1b
0345ffec 00000000 kernel32!BaseThreadStart+0×34

There is PrintDriverA module on the stack trace. Notice that we also have FilterException function on the stack trace. It raises the suspicion bar. We proceed to examining MessageBoxA parameters:

0:022> kv 100
ChildEBP RetAddr  Args to Child            
03456830 7739bf53 7739610a 00000000 00000000 ntdll!KiFastSystemCallRet
03456868 7738965e 186403ba 00000000 00000001 user32!NtUserWaitMessage+0xc
03456890 7739f762 77380000 05bdc880 00000000 user32!InternalDialogBox+0xd0
03456b50 7739f047 03456cac 00000000 ffffffff user32!SoftModalMessageBox+0x94b
03456ca0 7739eec9 03456cac 00000028 00000000 user32!MessageBoxWorker+0x2ba
03456cf8 773d7d0d 00000000 0ae7cc20 02639ea8 user32!MessageBoxTimeoutW+0x7a
03456d80 773c42c8 00000000 03456e14 03456df4 user32!MessageBoxTimeoutA+0x9c
03456da0 773c42a4 00000000 03456e14 03456df4 user32!MessageBoxExA+0x1b
03456dbc 6dfcf8c2 00000000 03456e14 03456df4 user32!MessageBoxA+0×45
034575f8 6dfd05cf 03456e5a 03457624 77bc6cd5 compstui!FilterException+0×174
[…]

0:022> da /c 90 03456e14
03456e14  “Function address 0×7c8100ca caused a protection fault. (exception code 0xc0000005). Some or all property page(s) may not be displayed.”

There was an exception indeed diagnosed by FilterException call. The exception is probably hidden somewhere on the raw stack:

0:022> !teb
TEB at 7ffde000
    ExceptionList:        03456d40
    StackBase:            03460000
    StackLimit:           03450000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffde000
    EnvironmentPointer:   00000000
    ClientId:             00000540 . 0000a8c4
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffd7000
    LastErrorValue:       0
    LastStatusValue:      c0000022
    Count Owned Locks:    0
    HardErrorMode:        0

0:022> dds 03450000 03460000
03450000  00000000
03450004  00000000
03450008  00000000
0345000c  00000000
[...]
03457674  03458574
03457678  7c8315c2 ntdll!RtlDispatchException+0×91
0345767c  03457710
03457680  03458574
03457684  0345772c
03457688  034576ec
0345768c  6dfd0a54 compstui!_except_handler3
03457690  00080000
03457694  03457710
03457698  0269b640
0345769c  026afc38
034576a0  00080000
034576a4  00080000
034576a8  026afc38
034576ac  026b2008
034576b0  034576cc
034576b4  7c82a771 ntdll!RtlpCoalesceFreeBlocks+0×383
034576b8  00000249
034576bc  026b2008
034576c0  00080000
034576c4  026afc38
034576c8  00080000
034576cc  00000000
034576d0  00080000
034576d4  034577b4
034576d8  7c82a90a ntdll!RtlFreeHeap+0×6b0
034576dc  00080608
034576e0  7c829f59 ntdll!RtlFreeHeap+0×70f
034576e4  03457870
034576e8  00000000
034576ec  00000001
034576f0  03460000
034576f4  006afc38
034576f8  03457c14
034576fc  7c82855e ntdll!KiUserExceptionDispatcher+0xe
03457700  03450000
03457704  0345772c
03457708  03457710
0345770c  0345772c
03457710  c0000005
03457714  00000000
03457718  00000000
0345771c  7c8100ca ntdll!RtlAllocateHeap+0×7b3
03457720  00000002
03457724  00000001
03457728  026c663c
0345772c  0001003f
03457730  00000000
03457734  00000000
03457738  00000000
0345773c  00000000
03457740  00000000
03457744  00000000
03457748  ffff027f
0345774c  ffff0000
[…]

It finally looks like a heap corruption:

0:022> .cxr 0345772c
eax=00000b28 ebx=00005e69 ecx=026c663c edx=0269b648 esi=0269b640 edi=00080000
eip=7c8100ca esp=034579f8 ebp=03457c14 iopl=0 nv up ei pl nz ac pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010216
ntdll!RtlAllocateHeap+0×7b3:
7c8100ca 8901  mov     dword ptr [ecx],eax  ds:0023:026c663c=????????

0:022> kL 100
ChildEBP RetAddr
03457c14 77c0b66f ntdll!RtlAllocateHeap+0×7b3
03457c44 77c1581a gdi32!EnumFontsInternalW+0×63
03457c68 32014246 gdi32!EnumFontFamiliesW+0×1c
03457ce4 32019ab4 PS5UI!BPackItemFontSubstTable+0×95
03457cf4 32014a0f PS5UI!BPackPrinterPropertyItems+0×19
03457d0c 32019e2b PS5UI!PPrepareDataForCommonUI+0×1af
0345813c 02118a57 PS5UI!DrvDevicePropertySheets+0×1dc
WARNING: Stack unwind information not available. Following frames may be wrong.
03458520 6dfd059a PrintDriverA!DrvDevicePropertySheets+0×3c7
03458584 6dfcff1e compstui!CallpfnPSUI+0xdb
034587f0 6dfd00a2 compstui!InsertPSUIPage+0×201
03458848 7307c9ae compstui!CPSUICallBack+0xed
03458870 6dfd059a winspool!DevicePropertySheets+0xd4
034588d4 6dfcff1e compstui!CallpfnPSUI+0xdb
03458b40 6dfd00a2 compstui!InsertPSUIPage+0×201
03458b98 6dfd06a3 compstui!CPSUICallBack+0xed
03458bcc 6dfd0799 compstui!DoCommonPropertySheetUI+0×74
03458be4 730801c5 compstui!CommonPropertySheetUIW+0×17
03458c2c 73080f5d winspool!CallCommonPropertySheetUI+0×43
03459074 35145947 winspool!PrinterPropertiesNative+0×10c
034590c4 3513a045 PrintDriverA!DllGetClassObject+0×159c7
0345e9ac 35131819 PrintDriverA!DllGetClassObject+0xa0c5
0345ebdc 32020661 PrintDriverA!DllGetClassObject+0×1899
0345ec04 3201b171 PS5UI!HComOEMPrinterEvent+0×33
0345ec48 02117a79 PS5UI!DrvPrinterEvent+0×239
0345eea4 7308218c PrintDriverA!DrvPrinterEvent+0xf9
0345eef0 761542cc winspool!SpoolerPrinterEventNative+0×57
0345ef0c 76155fd6 localspl!SplDriverEvent+0×21
0345ef30 76144799 localspl!PrinterDriverEvent+0×46
0345f3f8 76144ab2 localspl!SplAddPrinter+0×5f3
0345f424 74070193 localspl!LocalAddPrinterEx+0×2e
0345f874 7407025c spoolss!AddPrinterExW+0×151
0345f890 0100792d spoolss!AddPrinterW+0×17
0345f8ac 01006762 spoolsv!YAddPrinter+0×75
0345f8d0 77c80193 spoolsv!RpcAddPrinter+0×37
0345f8f8 77ce33e1 rpcrt4!Invoke+0×30
0345fcf8 77ce35c4 rpcrt4!NdrStubCall2+0×299
0345fd14 77c7ff7a rpcrt4!NdrServerCall2+0×19
0345fd48 77c8042d rpcrt4!DispatchToStubInCNoAvrf+0×38
0345fd9c 77c80353 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
0345fdc0 77c811dc rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
0345fdfc 77c812f0 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×42c
0345fe20 77c88678 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0345ff84 77c88792 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0345ff8c 77c8872d rpcrt4!RecvLotsaCallsWrapper+0xd
0345ffac 77c7b110 rpcrt4!BaseCachedThreadRoutine+0×9d
0345ffb8 77e64829 rpcrt4!ThreadStartRoutine+0×1b
0345ffec 00000000 kernel32!BaseThreadStart+0×34

lmt command shows many loaded print drivers but we advise the fans of driver elimination to remove or upgrade PrintDriverB and PrintDriveA. We also advise to enable full page heap on the spooler service to find the direct offender.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One common question is to how determine a service name from kernel memory dump where PEB information is not available (!peb). For example, there are plenty of svchost.exe processes running and one has a handle leak. I’m looking for a good simple method and in the mean time I suggested to use the following empirical data:

1. Look at driver modules on stack traces (e.g. termdd)

2. Look at the relative position of svchost.exe in the list of processes that reflects service startup dependency (!process 0 0)

3. Execution residue and string pointers on thread raw stacks (WinDbg script)

4. Process handle table (usually available for the current process according to my experience)

5. The number of threads and distribution of modules on thread stack traces (might require reference stack traces) 

6. IRP information (e.g. a driver, device and file objects), for example:

THREAD fffffa800c21fbb0  Cid 0264.4ba4  Teb: 000007fffff92000 Win32Thread: fffff900c2001d50 WAIT: (WrQueue) UserMode Alertable
    fffffa800673f330  QueueObject
IRP List:
fffffa800c388010: (0006,0478) Flags: 00060070  Mdl: 00000000
Not impersonating
DeviceMap                 fffff88000006160
Owning Process            fffffa8006796c10       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      30553196       Ticks: 1359 (0:00:00:21.200)
Context Switch Count      175424                 LargeStack
UserTime                  00:00:05.834
KernelTime                00:00:32.541
Win32 Start Address 0x0000000077a77cb0
Stack Init fffffa60154c6db0 Current fffffa60154c6820
Base fffffa60154c7000 Limit fffffa60154bf000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`154c6860 fffff800`01ab20fa nt!KiSwapContext+0x7f
fffffa60`154c69a0 fffff800`01ab55a4 nt!KiSwapThread+0x13a
fffffa60`154c6a10 fffff800`01d17427 nt!KeRemoveQueueEx+0x4b4
fffffa60`154c6ac0 fffff800`01ae465b nt!IoRemoveIoCompletion+0x47
fffffa60`154c6b40 fffff800`01aaf933 nt!NtWaitForWorkViaWorkerFactory+0x1fe
fffffa60`154c6c20 00000000`77aa857a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffffa60`154c6c20)
00000000`04e7fb58 00000000`00000000 0x77aa857a

3: kd> !irp fffffa800c388010
Irp is active with 6 stacks 6 is current (= 0xfffffa800c388248)
 No Mdl: System buffer=fffffa800871b210: Thread fffffa800c21fbb0:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

   Args: 00000000 00000000 00000000 00000000
>[  e, 0]   5  1 fffffa8006018060 fffffa8007bf0e60 00000000-00000000    pending
        \Driver\rdpdr
   Args: 00000100 00000000 00090004 00000000

3: kd> !fileobj fffffa8007bf0e60

\TSCLIENT\SCARD\14

Device Object: 0xfffffa8006018060   \Driver\rdpdr
Vpb is NULL
Access: Read Write SharedRead SharedWrite

Flags:  0x44000
 Cleanup Complete
 Handle Created

FsContext: 0xfffff8801807c010 FsContext2: 0xfffff8801807c370
CurrentByteOffset: 0
Cache Data:
  Section Object Pointers: fffffa800c50fdc8
  Shared Cache Map: 00000000

Any other ideas are appreciated. Please comment.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Source: Amazon DE (at the time of this writing)

It is also the top best seller among OpenTask titles: http://www.opentask.com/top-bestseller-april-2010

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’ve decided to adapt archaeological classificatory framework (using my favourite method of inquiry: metaphorical bijectionism) to lay out foundations for yet another attempt to classify DA+TA patterns):

Attribute  ↔ Pattern
Artefact   ↔ Component Artefact1
Assemblage ↔ Component Assemblage
Culture    ↔ Memory System Culture²

1 Can be either a component-generated artefact or a component like a module or symbol file
2 Typical examples of memory system cultures are Windows, UNIX or even “Multiplatform”

I propose a word Memoarchaeological for such a framework and Memoarchaeology for a branch of Memoretics that studies saved computer memory artifacts from past computations (as opposed to live memory).

Note: In one of the forthcoming issues of Debugged! MZ/PE magazine there will be presented yet another classificatory scheme.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Insufficient Memory (control blocks) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming CARE and STARE online systems additionally aim to provide software behaviour pattern identification via debugger log and trace analysis and suggest possible software troubleshooting patterns. The purpose of these post series is to provide high level overview of possible patterns of software behavior and how they can be recognised and analyzed. This work started in October, 2006 with the identification of computer memory patterns and later continued with software trace patterns. Bringing all of them under a unified linked framework seems quite natural to me.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Citrix Blog has published a short introduction:

Introducing the Citrix Support Twitter Team

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Insufficient Memory (physical memory) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -