Software Diagnostics Library

According to Google Analytics the number of visits / year increased by 4% since 2010 with almost 160,000 unique visitors (2% increase) from 180 countries and 34% of them are coming back. 2,725 visits were via 12 mobile operating systems (106% increase). Here are the top 100 network locations out of 28,932:

Top 25 visiting countries:

More than 8,000 portal and blog pages were viewed a total of more than 392,000 times with top 100 content pages:

More than 70,000 Google search keywords pointed to the portal and this blog with 100 most frequent (some are in Russian):

Special thanks to more  1,000 web sites that mention the portal and this blog with the first top 100:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is another tracing example of unified debugging patterns introduced previously.

- Analysis Patterns

Focus of Tracing

- Architectural Patterns

Debug Event Subscription / Notification

- Design Patterns

Shared Debug Event State

- Implementation Patterns

Shared Variable

- Usage Patterns

Saving a memory address obtained at a breakpoint event in a debugger pseudo-register for use at later breakpoint events

Debugging.tv published a case study for tracing window messages in WinDbg. There a pseudo-register is used to save a buffer address before GetMessage call and then to reuse it after the call. Please look at Event State Management slide on Frames episode 0×06 presentation. The full WinDbg log and the recording are available there too.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Based on ideas of Roman Jakobson about “marked” and “unmarked” categories we propose another pattern called Marked Message that groups trace messages based on having some feature or property. For example, marked messages may point to some domain of software activity such related to functional requirements and therefore may help in troubleshooting and debugging. Unmarked messages include all other messages that don’t say anything about such activities (although may include messages pointing to such activities indirectly we unaware of) or messages that say explicitly that no such activity has occurred. We can even borrow a notation of distinctive features from phonology and annotate any trace or log after analysis to compare it with a master trace, for example, compose the following list of software trace distinctive features:

session database queries [+]
session initialization [-]
socket activity [+]
process A launched [+]
process B launched [-]
process A exited [-]
[…]

Here [+] means the activity is present in the trace and [-]  means the activity is either undetected or definitely not present. Sometime a non-present activity can be a marked activity corresponding to all inclusive unmarked present activity (see, for example, No Activity pattern).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I created a special picture based on CPU and memory timing diagram (an optimistic version of the original computicart):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

150 bugtations so far…

Program history has two sides, a computational and a human.

Philip Schaff

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is the first initiative for the year of software trace analysis: the first and unique software trace and log analysis training based entirely on patterns of software behavior. No longer you will be frustrated when opening a software trace with millions of messages from hundreds of software components, threads and processes.

Memory Dump Analysis Services (DumpAnalysis.com) organizes a training course:

Learn how to efficiently and effectively analyze software traces and logs from complex software environments. Covered popular software logs and trace formats from Microsoft and Citrix products and tools including Event Tracing for Windows (ETW) and Citrix Common Diagnostics Format (CDF). Learn how to use pioneering and innovative pattern-driven software problem behavior analysis to troubleshoot and debug software incidents.

If your are registered you are allowed to optionally submit your software traces and logs before the training. This will allow us in addition to the carefully constructed problems tailor additional examples to the needs of the attendees.

The training consists of 2 two-hour sessions and additional homework exercises. When you finish the training you additionally get:

1. A full transcript in PDF format (retail price $200)
2. 6 volumes of Memory Dump Analysis Anthology in PDF format (retail price $120)
3. A personalized attendance certificate with unique CID (PDF format)
4. Free Dump Analysis World Network membership including updates to full PDF transcript Q&A section

Prerequisites: Basic Windows troubleshooting.

Audience: Software technical support and escalation engineers, software maintenance engineers, system administrators.

Session 1: October 12, 2012 4:00 PM - 6:00 PM BST
Session 2: October 15, 2012 4:00 PM - 6:00 PM BST

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/r/5287623225237732608

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Making Software A Better World.

Dmitry Vostokov, Memory Dump Analysis Services

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The number of software trace analysis patterns approaches the critical mass of 50 and we have decided to focus on software tracing and logging in the forthcoming year. Some books on tracing including Volume 7 of Memory Dump Analysis Anthology will be published by OpenTask during that year and our efforts will be to further advance software narratology, software trace linguistics, and software trace analysis in the context of memory dump analysis, generative debugging and modeling software behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

More than 4 years passed since I provided a longer structuralist definition. Recently I came to recognize a pattern-driven iterative and incremental nature of memory and software trace analysis and post-construction software problem solving in general and therefore a one sentence definition became necessary:

“Recognition and interpretation of patterns of software behavior”

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Frequently we observe that several trace messages form a single semantic unit we call Macrofunction where individual trace messages serve the role of microfunctions. We borrowed this idea and distinction from functionalist linguistics. An example would be a software trace fragment where messages log an attempt to update a database:

#     Module   PID  TID   Time         Message
[...]
42582 DBClient 5492 9476  11:04:33.398 Opening connection
[...]
42585 DBClient 5492 9476  11:04:33.398 Sending SQL command
[...]
42589 DBServer 6480 10288 11:04:33.399 Executing SQL command
[...]
42592 DBClient 5492 9476  11:04:33.400 Closing connection
[...]

Please note that these macrofunctions need not be from the same ATID in the traditional sense like in the example above unless we form adjoint threads from certain fragments like “DB”.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the new initiatives for 2012 is the development of SPDL (Software Problem Description Language). Its purpose is automatic generation of a software troubleshooting tool(s) based on the description of a problem. Here software problem means a post-construction problem as outlined in Introduction to Pattern-Driven Software Problem Solving. The tool construction will utilize an expanded set of DebugWare and Workaround patterns together with the refind version of RADII software development process. This will also provide necessary effectiveness, efficiency and enhanced problem solving capabilities to existing TaaS (Tools as a Service) implementations that are limited in the number of tools they offer.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:

• 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
• 4 new pattern interaction case studies
• 11 new trace analysis patterns
• New Debugware pattern
• Introduction to UI problem analysis patterns
• Introduction to intelligence analysis patterns
• Introduction to unified debugging pattern language
• Introduction to generative debugging, metadefect template library and DNA of software behaviour
• The new school of debugging and trends
• .NET memory dump analysis checklist
• Software trace analysis checklist
• Introduction to close and deconstructive readings of a software trace
• Memory dump analysis compass
• Computical and Stack Trace Art
• The abductive reasoning of Philip Marlowe
• Orbifold memory space and cloud computing
• Memory worldview
• Interpretation of cyberspace
• Relationship of memory dumps to religion
• Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5

Product information:

• Title: Memory Dump Analysis Anthology, Volume 6
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 300 pages
• Publisher: Opentask (December 2011)
• ISBN-13: 978-1-908043-19-1

• Hardcover: 300 pages
• Publisher: Opentask (January 2012)
• ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have Linked Messages through some common parameter or attribute. One such example can be found in ETW traces related to kernel process creation notifications. Here we got adjoint thread for module PIDNotify:

#      Module    PID  TID  Time         Message
[...]
128762 PIDNotify 1260 6208 15:53:15.691 Create: ParentID 0x000004EC PID 0×000018D4
[…]
128785 PIDNotify 6356 6388 15:53:15.693 Load: ImageName \Device\HarddiskVolume1\Windows\System32\abscript.exe PID 0×000018D4
[…]
131137 PIDNotify 6356 4568 15:53:15.936 Create: ParentID 0×000018D4 PID 0×00001888
[…]
131239 PIDNotify 6280 6376 15:53:15.958 Load: ImageName \Device\HarddiskVolume1\Windows\System32\wscript.exe PID 0×00001888
[…]
132899 PIDNotify 6356 5704 15:53:16.462 Create: ParentID 0×000018D4 PID 0×00001FD0
[…]
132906 PIDNotify 8144 7900 15:53:16.464 Load: ImageName \Device\HarddiskVolume1\Windows\System32\cmd.exe PID 0×00001FD0
[…]

We see that messages 128762 and 128785 are linked through PID parameter and linked to messages 131137 and 132899 through PID - ParentID parameter relationship. Similar linkages exist for messages 131137 / 131239 and 132899 / 132906.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

switch(years_at_citrix)
{
  case 5:
  write_blog_post(”I’ve just passed 5 year mark … “);
  wait_for_certificate();
  write_blog_post(”Shortly after celebrating 5 years … “);
  break;
  case 6:
  write_blog_post(”Threads in my process run very fast. Not long ago … “);
  break;
  case 7:
  write_blog_post(”Transition to kernel mode and space … “);
  break;
  case 8:
  write_blog_posts(”A byte has passed”, “8 bits of Citrix”, … “);
  break;
  case 9:
  // … TBD
}

No default case label in the code yet.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoretics promotes pattern-driven memory dump and software trace analysis which has many uses but not limited to:

• Software and site reliability
• Software Debugging
• QA and Software Testing
• Computer Security
• Software Troubleshooting
• Malware Research and Analysis
• Tools as a Service (TaaS)
• Supportability
• Software Diagnostics

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoretics as a science of memory snapshots borrows many ideas from the following disciplines (the list is not exhaustive):

• Troubleshooting and Debugging
• Intelligence Analysis
• Critical Thinking
• Forensics
• Linguistics
• Archaeology
• Psychoanalysis
• History
• Mathematics: Sets and Categories
• Literary Criticism and Narratology

It also contributes many ideas back. The following diagram depicts such an interaction:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Original Message pattern deals with software trace messages where certain activity is repeated several times but the only the first message occurrence or specific message vocabulary has significance for analysis activity. Typical example from CDF/ETW tracing is module load events:

#     Module  PID   TID   Time         Message
[...]
35835 ModuleA 11000 11640 17:27:28.720 LoadImageEvent: \Device\HarddiskVolume2\Windows\System32\userinit.exe PId 5208
[…]
37684 ModuleA 12332 9576  17:27:29.063 LoadImageEvent: \Windows\System32\userinit.exe PId 573C
[…]
37687 ModuleA 12332 9576  17:27:29.064 LoadImageEvent: \Windows\System32\userinit.exe PId 573C
[…]

What we are looking here is for message invariant like “.exe” but interested in the occurrence of specific path structures like \Device\HarddiskVolume because in our troubleshooting context they signify process launch sequence during terminal session initialization.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

UI Message pattern is very useful for troubleshooting system-wide issues because we can map visual behaviour to various activity regions and consider such messages as significant events.

#    Module  PID  TID  Time         Message
[...]
2782 ModuleA 2124 5648 10:58:03.356 CreateWindow: Title "..." Class "..."
[...]
3512 ModuleA 2124 5648 10:58:08.154 Menu command: Save Data
[...]
3583 ModuleA 2124 5648 10:58:08.155 CreateWindow: Title "Save As" Class "Dialog"
[... Data update and replication related messages ...]
4483 ModuleA 2124 5648 10:58:12.342 DestroyWindow: Title "Save As" Class "Dialog"
[...]

By filtering the emitting module we can create an adjoint thread:

#    Module  PID  TID  Time         Message
[...]
2782 ModuleA 2124 5648 10:58:03.356 CreateWindow: Title "..." Class "..."
3512 ModuleA 2124 5648 10:58:08.154 Menu command: Save Data
3583 ModuleA 2124 5648 10:58:08.155 CreateWindow: Title "Save As" Class "Dialog"
4483 ModuleA 2124 5648 10:58:12.342 DestroyWindow: Title "Save As" Class "Dialog"
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Second Eye (or sometimes a stronger variant “second pair of eyes”) - another engineer you typically need when you don’t see anything useful in a memory dump, software trace or source code for problem resolution purposes. You are anxious to recommend something useful.

Examples: Don’t see anything in this huge trace. I need a second eye.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was inspired today for a new comic art movement called Computicart (Computical Art). The first composition is called Cloud Traces (remember that memory dumps are just bigger software traces and software traces are just smaller memory dumps):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -