Crash Dump Analysis

Debugging Experts Magazine Online (online version of Debugged! MZ/PE) underwent the total redesign:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The magazine issue features my article on adjoint threads, the first part of a long article on Windows thread classification, a comparison article on Citrix CDF analysis tools and a review of Advanced .NET Debugging book.

The issue can be found on either www.debuggingexpert.com or www.debuggingexperts.com:

http://www.debuggingexperts.com/debugged-march-10

The print issue will be available in September with the back cover featuring the summary of WinDbg multithreading commands.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free webinar

Date: 18th of August 2010
Time: 21:00 (BST) 16:00 (Eastern) 13:00 (Pacific)
Duration: 90 minutes

Topics include:

- User vs. kernel vs. physical (complete) memory space
- Challenges of complete memory dump analysis
- Common WinDbg commands
- Patterns
- Common mistakes
- Fiber bundles
- Hands-on exercise: a complete memory dump analysis
- A guide to DumpAnalysis.org case studies

Prerequisites: working knowledge of basic user process and kernel memory dump analysis or live debugging using WinDbg 

The webinar link will be posted before 18th of August on DumpAnalysis.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Our future sponsor has been registered in Ireland and has its own independent website and logo: DumpAnalysis.com

More information will be available later this month.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Being a software engineer, the author penetrated a software technical support department of a major software company rising to a management position. There he started collecting various management bits and tips promising everyone to write a management book. After moving back to engineering he became a director of several software research, education, publishing and software behavior analysis consultancy institutions including a museum. This book is an anthology of selected and edited blog posts from his Management Bits and Tips blog.

What this book has to do with the crash dump analysis then? Considering metaphorically an organization as a software machine, teams as processes and individuals as threads the author had applied his unique knowledge of software crashes and hangs to organizational project failures.

• Title: Management Bits: An Anthology from Reductionist Manager
• Author: Dmitry Vostokov
• Publisher: OpenTask (September 2010)
• Language: English
• Product Dimensions: 19.8 x 12.9
• Paperback: 100 pages
• ISBN-13: 978-1906717131

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The rules of the previously announced competition have been changed. You can now also nominate someone by sending a debugging story link. Please send your story or a nomination using this page: http://www.dumpanalysis.org/contact or dmitry dot vostokov at dumpanalysis dot org (if a story is with pictures).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Adding AI. Analysis Improvement.

After reading earlier today Windows Internals pages about system audit an idea came to my mind in the evening to provide audit services for memory dump and software trace analysis. One mind is good but two are better, especially if the second is a pattern-driven AI. Here are possible problem scenarios:

Problem: You are not satisfied with a crash report.

Problem: Your critical issue is escalated to the VP level. Engineers analyze memory dumps and software traces. No definite conclusion so far. You want to be sure that nothing has been omitted from the analysis.

Problem: You analyze a system dump or a software trace. You need a second pair of eyes but don’t want to send your memory dump due to your company security policies.

Other scenarios (use cases) will be added as soon as I see the service fit to the realities of software technical support.

I plan to make this service operational in July - August, 2010. Prices to be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

By analogy with paratext let’s introduce a software narratological concept of the extended software trace that consists of a software trace plus additional supporting information that makes troubleshooting and debugging easier. Such “paratextual” information can consists of pictures, videos, accounts of scenarios and past problem histories, customer interviews and even software trace delivery medium and format (if preformatted).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally Citrix has published a tool (written by my colleague Colm Naish, lead escalation engineer) that allows controlled injection of events into CDF (ETW) trace message stream. This is useful in many troubleshooting scenarios where we need to rely on Significant Event and Anchor Message analysis patterns to partition traces into artificial Activity Regions to start our analysis with. This is also analogous for the imposition of the external time on the stream of tracing events from software narratology perspective:

CDFMarker On Demand - For XenApp and XenDesktop

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

My drive to generalization led me to place an adornment on the portal to highlight the fact that memory and software trace analysis patterns are under an umbrella of general software behaviour patterns:

http://www.dumpanalysis.org/Software-Behavior-Patterns-Headline

In the forthcoming post series I plan to write about similarities between these two branches and also provide pattern examples from non-Windows platforms. All this material will provide the foundation for the forthcoming book Software Behavior: A Guide to Systematic Analysis (ISBN: 978-1906717162).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Befind every trace and its messages is source code:

Borrowing the acronym PLOT (Program Lines of Trace) we now try to discern basic source code patterns that give rise to simple message patterns in software traces. There are only a few distinct PLOTs and the ability to mentally map trace statements to source code is crucial to software trace reading and comprehension. More about that in subsequent parts. More complex message patterns (for example, specific message blocks or correlated messages) arise from supportable and maintainable realizations of architectural, design and implementation patterns and will be covered in another post series.

I was thinking about acronym SLOT (Source Lines of Trace) but decided to use PLOT because it metaphorically bijects into literary theory and narrative plots.

Forthcoming CDF and ETW Software Trace Analysis: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

PLOT - Program Lines of Trace - the source code lines behind trace messages

Examples: What a plot do we have here! The struggle against the monster database component and endless voyages across space boundaries.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

Modern pattern-driven software trace analysis on Microsoft and Citrix platforms urgently requires a practical guide and OpenTask plans to publish this summer the following book in both Practical Foundations and Systematic Software Fault Analysis series:

• Title: Citrix Common Diagnostic Facility (CDF) and Microsoft Event Tracing for Windows (ETW) Software Trace Analysis: Practical Foundations
• Author: Dmitry Vostokov
• Publisher: Opentask (August 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN: 1906717176
• ISBN-13: 978-1906717179
• Paperback: 200 pages

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

c’t – Magazin für Computertechnik has published a review of First Fault Software Problem Solving book: 

http://www.heise.de/ct/inhalt/2010/08/192/ (in German)

Fabian Röken kindly translated it into English:

No single large software package comes without errors. It seems that customers simply accept this, patiently waiting and hoping for patches or updates. Skwire sticks up for a more target-aimed approach: one will never get a faultless software, but it would already be a great improvement if flaws were already solved on their first occurrence (”first fault”) and not only after a long analysis (”second fault”).

The advantages are actually obvious. However, a corresponding stringent system architecture, as common on mainframes such as IBM’s z/OS, did not become prevalent in the PC market.

Skwire outlines the types of errors and strategies to resolve them in all details. His 40 years of experience, such as at IBM, shimmers through again and again. He puts emphasis on making sure that the reader understands the terminology he is using: “What is a problem in the first place?”, “What is a service point?” - in some cases he also explains specific metrics such as the “serviceability rating”.

His tool classification includes teaching tips, e.g. regarding the structure of a protocol in case of errors; or for tracking the important information how often an error must occur before a solution has to be approached. His suggestions equally address developers, designers, testers, managers - and the end user. In his last chapter he presents and reviews commercial tools in the first fault and second fault environment.

Skwire addresses a topic which is unfortunately very much neglected, and this alone already makes it worth enough to take a look at his book (***). Short quotations and humorous drawings relax the technical topic. If you are looking for an overview then you will be fine with this book. However, if you are a software developer looking for source code samples then you will search in vain. Skwire has released the book under the print-on-demand process. You will find it on Amazon, for example.

(Tobias Engler/fm)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming CARE and STARE online systems additionally aim to provide software behaviour pattern identification via debugger log and trace analysis and suggest possible software troubleshooting patterns. The purpose of these post series is to provide high level overview of possible patterns of software behavior and how they can be recognised and analyzed. This work started in October, 2006 with the identification of computer memory patterns and later continued with software trace patterns. Bringing all of them under a unified linked framework seems quite natural to me.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The following tool published by Citrix follows DebugWare patterns in its overall architecture and design and was implemented by a team of engineers using RADII process:

SsOnExpert - Single Sign-On XenApp Plug-in Troubleshooting Tool

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Time flows fast and I have refined my code I wrote at 6 year anniversary to include finer monthly tracing:

switch (months_at_citrix)
{
  case 60:
    write_blog_post(”I’ve just passed 5 year mark … “);
    wait_for_certificate(); 
    write_blog_post(”Shortly after celebrating 5 years … “);
    break;
  case 72:
    write_blog_post(”Threads in my process run very fast. Not long ago … “);
    break;
  case 78:
    write_blog_post(”Time flows fast and I have refined my code …“);
    break;
case 84:
    // … TBD 
}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

.SYS - Sponsor YourSelf or Sponsor YourSelves.

Examples: I’m developing a fantastic project.SYS

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):

• Are you interested in on-site training, prefer traveling or attending webinars?
• Are you interested in software trace analysis as well?
• What specific topics are you interested in?
• What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)

Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.

Looking forward to your responses. Any suggestions are welcome.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -