Crash Dump Analysis

The magazine issue features my article on adjoint threads, the first part of a long article on Windows thread classification, a comparison article on Citrix CDF analysis tools and a review of Advanced .NET Debugging book.

The issue can be found on either www.debuggingexpert.com or www.debuggingexperts.com:

http://www.debuggingexperts.com/debugged-march-10

The print issue will be available in September with the back cover featuring the summary of WinDbg multithreading commands.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free webinar

Date: 18th of August 2010
Time: 21:00 (BST) 16:00 (Eastern) 13:00 (Pacific)
Duration: 90 minutes

Topics include:

- User vs. kernel vs. physical (complete) memory space
- Challenges of complete memory dump analysis
- Common WinDbg commands
- Patterns
- Common mistakes
- Fiber bundles
- Hands-on exercise: a complete memory dump analysis
- A guide to DumpAnalysis.org case studies

Prerequisites: working knowledge of basic user process and kernel memory dump analysis or live debugging using WinDbg 

The webinar link will be posted before 18th of August on DumpAnalysis.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Our future sponsor has been registered in Ireland and has its own independent website and logo: DumpAnalysis.com

More information will be available later this month.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was looking for Wordpress plugins to display past year (two-, three-, etc.) old posts and after a few tries decided to create such posts manually to add fresh perspective and new comments on them (in italics). Today we start with June 21st.

2009:

• Debugalov has been burnt! - The book still sells every month and I’m now thinking about a hardcover gift edition. The cover fascinates many people, see for example this review: Book Review - The Adventures of Dr Debugalov. Now we also have Dr. DebugLove. Who is a good and who is a bad guy? Or is it a personality split?

2008:

No dumps on that day

2007:

Looks like the very prolific day. There were 3 blog posts:

• Crash Dump Analysis Patterns (Part 16a) - Stack overflow in kernel. Generated some comments and can also be seen in the following pattern case study: Lateral damage, stack overflow and execution residue

• Repair Clipboard Chain 2.0.1 - One of the most popular Citrix tool in the past

• Guessing stack trace - This old command still works for x86 WinDbg and x86 memory dumps

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Adding AI. Analysis Improvement.

After reading earlier today Windows Internals pages about system audit an idea came to my mind in the evening to provide audit services for memory dump and software trace analysis. One mind is good but two are better, especially if the second is a pattern-driven AI. Here are possible problem scenarios:

Problem: You are not satisfied with a crash report.

Problem: Your critical issue is escalated to the VP level. Engineers analyze memory dumps and software traces. No definite conclusion so far. You want to be sure that nothing has been omitted from the analysis.

Problem: You analyze a system dump or a software trace. You need a second pair of eyes but don’t want to send your memory dump due to your company security policies.

Other scenarios (use cases) will be added as soon as I see the service fit to the realities of software technical support.

I plan to make this service operational in July - August, 2010. Prices to be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This paleodebugging tool was excavated from Central Russia (thanks to Mr. Kutuzov) and generously provided for a photo session by its owner Mr. Mansour:

It also inspired this sequence of strcat: Analog -> Anatrace -> Analyzer -> Tracelyzer -> Loglyzer.

… enough tracing. It’s time to close our session:

… what is left? If you are curious, look at this conceptual picture:

Component Trace

If you wonder what electricity has to do with tracing (at a metaphorical level) look at this trace analysis pattern:

Statement Density and Current

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally, after careful consideration, I’ve come up with the topic that has been neglected so far but at the same time important for both camps (kernel space and user space, including managed space): complete memory dump and software trace analysis. I plan to publish the first webinar agenda early in July and deliver the webinar in August (the date should be finalized by mid July).

PS. Sailing memory spaces under an RGB flag

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally Citrix has published a tool (written by my colleague Colm Naish, lead escalation engineer) that allows controlled injection of events into CDF (ETW) trace message stream. This is useful in many troubleshooting scenarios where we need to rely on Significant Event and Anchor Message analysis patterns to partition traces into artificial Activity Regions to start our analysis with. This is also analogous for the imposition of the external time on the stream of tracing events from software narratology perspective:

CDFMarker On Demand - For XenApp and XenDesktop

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Some recent news about StressPrinters tool designed according to Tool Façade DebugWare pattern:

“HP tests its print drivers with the StressPrinters tool provided by Citrix to simulate a user logon where multiple printers are autocreated concurrently.”

Source

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Modern pattern-driven software trace analysis on Microsoft and Citrix platforms urgently requires a practical guide and OpenTask plans to publish this summer the following book in both Practical Foundations and Systematic Software Fault Analysis series:

• Title: Citrix Common Diagnostic Facility (CDF) and Microsoft Event Tracing for Windows (ETW) Software Trace Analysis: Practical Foundations
• Author: Dmitry Vostokov
• Publisher: Opentask (August 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN: 1906717176
• ISBN-13: 978-1906717179
• Paperback: 200 pages

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It is time to start being systematic. In addition to all-encompassing multi-volume Memory Dump Analysis Anthology OpenTask starts Systematic Software Fault Analysis series with Crash Dump Analysis: Practical Foundations as the first book. It introduces basic definitions, tools, memory dump collection and preliminary analysis methods for Windows platforms including legacy versions. This practical reference guide is a must have for system administrators of Windows server platforms and client workstations, technical support engineers and general Windows users. It builds foundation for the second book Crash Dump Analysis for System Administrators and Support Engineers and the remaining tetralogy books Windows Crash Dump Analysis and Advanced Windows Crash Dump Analysis.

Product information:

• Title: Crash Dump Analysis: Practical Foundations (Windows Edition, Systematic Software Fault Analysis Series)
• Authors: Dmitry Vostokov
• Publisher: Opentask (May 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-98-8
• Paperback: 100 pages

Fromt cover:

Table of Contents to be published soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Yesterday I discovered the blog j00ru//vx where I was pleased to see another memory visualization approach which I classify as synthetic:

x86 Kernel Memory Space Visualization (KernelMAP v0.0.1)

So far now I put a bit more extended (but in no way complete) classification with links (based on my previous blog post where every category is presented in chronological order of my encounter with links):

1. Synthetic

• Numerous debuggers that show memory maps for easy navigation (links to follow)
• VMMap
• KernelMAP v0.0.1
• MemSpyy
• Virtual memory usage and GC heap usage

2. Natural

a. Static

• Dump2Picture (Windows)
• dump2pic (Linux)
• 2D and 3D visualization using general-purpose tools like ParaView.

b. Semi-dynamic

• WinDbg scripts

c. Dynamic 

• Haywire

Please let me know any other approaches or links you know. 

PS. I’m currently a big fan of artificial evolution and recommend this fantastic full-color book that has good ideas about expression-based visualization:

The Art of Artificial Evolution: A Handbook on Evolutionary Art and Music (Natural Computing Series)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The following tool published by Citrix follows DebugWare patterns in its overall architecture and design and was implemented by a team of engineers using RADII process:

SsOnExpert - Single Sign-On XenApp Plug-in Troubleshooting Tool

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Do you see my thread of thought? Tender ≈ easily crushed, so a tender button can easily crash or can be easily cr(a)ushed. When I saw the title of a book “Tender Buttons” I immediately recalled TestDefaultDebugger and similar programs. Apartment is from COM lexicon.

Tender buttons that crash: objects, messages, apartments.

Gertrude Stein, The Tender buttons: objects, food, rooms

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

.SYS - Sponsor YourSelf or Sponsor YourSelves.

Examples: I’m developing a fantastic project.SYS

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):

• Are you interested in on-site training, prefer traveling or attending webinars?
• Are you interested in software trace analysis as well?
• What specific topics are you interested in?
• What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)

Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.

Looking forward to your responses. Any suggestions are welcome.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Metaphorical bijection from literary narratology to software narratology provides a pattern of Background and Foreground Components. This can be easily illustrated on pseudo-trace color diagrams. Suppose we troubleshoot a graphical issue using an ETW trace containing the output from all components of the problem system. Graphic components and their messages are foreground for a trace viewer (a person) against numerous background components (for example, database, file and registry access, shown in shades of green):

Trace viewers (for example, CDFAnalyzer) can filter out background component messages and present only foreground components (that I propose to call component foregrounding):

Of cource, this process is iterative and parts of what once was foreground becomes background and candidate for further filtering:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a new open source tool similar to PDBFinder in functionality but with much simpler interface and internal implementation (based on the file name structure of the certain classes of TMF files for ETW). To be released this month on TraceAnalysis.org.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -