Software Diagnostics Library

Some Dia|gram language illustrations of trace and log analysis patterns can be easily converted to more usual graphs. For example, Trace Field can be converted to Combed Trace or Adjoint Threads of Activity:

The resulted series can be plotted as a graph:

If time is values are not uniform Time Scale analysis pattern may be involved with the constructed Motivic Trace. We call this analysis pattern Trace Graph.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If we take Combed Trace for Thread of Activity or some Adjoint Thread of Activity, strip other message content, and then trace all non-empty values we get Trace Contour:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A typical software trace or log (for example, from Process Monitor) lists messages from several processes and threads sequentially. However, such columns may be split into individual process ID or thread ID columns. The same can be done for any Adjoint Thread and illustrated in the following diagram:

We call this analysis pattern Combed Trace by analogy with multibraiding.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In Adjoint Message analysis pattern description we mentioned compressing message sequences having the same message attribute into one message. Considering the trace as “topological” space and message attribute as “equivalence” relation we introduce Quotient Trace analysis pattern by analogy with quotient space in topology. By endowing message sequences having the same attribute with some “metric” such as cardinality of Message Set we can also visually distinguish resulted quotient messages if they have the same attribute but from different sequences at different times. All this is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

By analogy with Adjoint Thread of Activity we introduce Adjoint Message analysis pattern. Most if not all analysis patterns focus on log message text and consider TID, PID, Module, source file and function as its attributes. However, we can choose one of attributes and consider it as a message in its own right with the original message text consigned now as another attribute. Then we can analyze the structure of the trace from the perspective of that newly selected message:

Since the number of different message values now is smaller (for example, module names) compared to normal trace messages we can use them in protein-like encoding and structure analysis schemes (see Software Trace and Logs as Proteins). We metaphorically name Adjoint Messages as Amino-acid-Messages (A-Messages). We can also compress same message sequences into one message which may be useful for pattern matching (and even use different color intensities to represent message cardinalities):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

General traces and logs may have Message Space regions “surrounded” by the so-called Interspace. Such Interspace regions may link individual Message Space regions like in this diagram generalizing WinDbg !process 0 3f command output:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Message stream can be considered as a union of Message Spaces. A message space is an ordered set of messages preserving the structure of the overall trace. Such messages may be selected based on a memory space they came from or can be selected by some other general attribute, or a combination of attributes and facts. The differences from Message Set is that Message Space is usually much larger (with large scale structure) with various Message Sets extracted from it later for fine grained analysis. This pattern also fits nicely with Adjoint Spaces. Here’s an example of kernel and managed spaces in the same CDF / ETW trace from Windows platform where we see that kernel space messages came not only from System process but also from other process contexts:

In the context of general traces and logs such as debugger logs separate Message Space regions may be linked (or “surrounded”) by Interspace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Reading Boris Uspensky’s book “A Poetics of Composition: The Structure of the Artistic Text and Typology of a Compositional Form” (in its original Russian version) led me to borrow the concept of viewpoints. The resulting analysis pattern is called Trace Viewpoints. These viewpoints are, “subjective” (semantically laden from the perspective of a trace and log reader), and can be (not limited to):

- Error viewpoints (see also False Positive Error, Periodic Error, and Error Distribution)

- Use case (functional) viewpoints (see also Use Case Trail)

- Architectural (design) viewpoints (see also Milestones)

- Implementation viewpoints (see also Implementation Discourse, Macrofunctions, and Focus of Tracing)

- Non-functional viewpoints (see also Counter Value and Diegetic Messages)

- Signal / noise viewpoints (see also Background and Foreground Components)

In comparison, Activity Regions, Data Flow, Thread of Activity, and Adjoint Thread of Activity are “objective” (structural, syntactical) viewpoints.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

So far we have been discussing trace analysis patterns related to execution of a particular software version. However, software code changes and also its tracing and logging output: from large scale changes where components are replaced to small scale code refactoring affecting message structure and format. On a software narratological level this corresponds to a narrative about a software trace or log, it evolution. Such Meta Trace analysis pattern is different from Master Trace pattern where the latter is similar to what Metanarrative is usually meant in narratology: a master or grand idea - an expected trace if all functional requirements were correctly identified and implemented during software construction and non-functional ones are met during software execution.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Whereas, when naturally visualized with Dump2Picture, a CDF trace looks like a sea with embedded library, a process monitor log looks more like a fragment from a memory dump with some large scale internal structure:

Its 450×1280 slice:

Top and bottom of x4 magnified image:

…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The modern software trace recording, visualization and analysis tools such as Process Monitor, Xperf, WPR and WPA provide stack traces associated with trace messages. Consider stack traces as software traces we have, in a more general case, traces (fibers) bundled together on (attached to) a base software trace. For example, a trace message, that mentions an IRP can have its I/O stack attached together with thread stack trace with function calls leading to a function that emitted the trace message. Another example is association of different types of traces with trace messages such as managed and unmanaged ones. This general trace analysis pattern needs a name so we opted for Fiber Bundle as analogy with a fiber bundle from mathematics. Here’s a graphical representation of stack traces recorded for each trace message where one message also has an associated I/O stack trace:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When depicting trace analysis patterns I used two-dimensional diagrams based on CDF (ETW) traces such as this one for a bifurcation point:

While working today on a new pattern I needed a new expressive way to graphically illustrate the same idea of trace bifurcation points but without too much drawing. Traces from particle chambers and scattering diagrams came to my imagination after draw the first few diagrams illustrating bifurcation points:

Time directional arrow end can be omitted:

Trace variation after a bifurcation point can be indicated by angle partition:

The case when a variation also happens before is illustrated on this diagram:

and the case with several bifurcations:

Are N-bifurcations like on the diagram below possible?

Yes, they are, if the course of execution depends on some non-binary trace message parameter such as a loaded module that implements a required interface differently.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Counter Value pattern covers performance monitoring and its logs. A counter value is some variable in memory, for example, a module variable, that is updated periodically to reflect some aspect of state or it can be calculated from different such variables and presented in trace messages. Such messages can also be organized in a similar format as ETW based traces we usually consider as examples for our trace patterns:

Source  PID TID   Function         Value
=================================================
[…]
System    0   0   Committed Memory 12,002,234,654
Process 844   0   Private Bytes    345,206,456
System    0   0   Committed Memory 12,002,236,654
Process 844   0   Working Set      122,160,068
[…]

Therefore, all other trace patterns such as adjoint thread (can be visualized via different colors on a graph), focus of tracing, characteristic message block (for graphs), activity region, significant event, and others can be applicable here. There are also some specific patterns such as global monotonicity and constant value that we discuss with examples in subsequent parts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After writing about music for debugging and founding software narratology I decided to start writing about fiction. The first masterpiece is The Sound and The Fury by William Faulkner. I confess that I’m in love with Folio Society books and when I saw this color version (an original idea by Faulkner now fulfilled by modern printing technology) I immediately recognized its importance for software trace analysis:

http://www.foliosociety.com/book/SAF/sound-and-the-fury

I’m pretty sure Faulkner would have been delighted to see trace analysis patterns and how they may help in writing fiction.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In a week this short full color book should appear in online bookstores:

• Title: Software Narratology: An Introduction to the Applied Science of Software Stories
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Description: This is a transcript of Memory Dump Analysis Services Webinar about Software Narratology: an exciting new discipline and a field of research founded by DumpAnalysis.org. When software executes it gives us its stories in the form of UI events, software traces and logs. Such stories can be analyzed for their structure and patterns for troubleshooting, debugging and problem resolution purposes. Topics also include software narremes and their types, anticipatory software construction and software diagnostics.
• Publisher: OpenTask (April 2012)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 26 pages
• ISBN-13: 978-1908043078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free Webinar about software narratology: an application of ideas from narratology to software narrative stories.

Learn about this exciting new discipline and a field of research. When software executes it gives us its stories in the form of UI events, software traces and logs. Such stories can be analyzed for their structure and patterns for troubleshooting, debugging and problem resolution purposes.

Prerequisites: basic software troubleshooting and ability to read software generated logs.

Audience: anyone can benefit from this new perspective: from software users to software engineers and managers.

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/603136294

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The number of software trace analysis patterns approaches the critical mass of 50 and we have decided to focus on software tracing and logging in the forthcoming year. Some books on tracing including Volume 7 of Memory Dump Analysis Anthology will be published by OpenTask during that year and our efforts will be to further advance software narratology, software trace linguistics, and software trace analysis in the context of memory dump analysis, generative debugging and modeling software behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoretics as a science of memory snapshots borrows many ideas from the following disciplines (the list is not exhaustive):

• Troubleshooting and Debugging
• Intelligence Analysis
• Critical Thinking
• Forensics
• Linguistics
• Archaeology
• Psychoanalysis
• History
• Mathematics: Sets and Categories
• Literary Criticism and Narratology

It also contributes many ideas back. The following diagram depicts such an interaction:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 5 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+5

It features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Its table of contents is available here:

www.dumpanalysis.org/MDAA/MDA-Anthology-V5-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 6 that should be available in November-December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

With the new year starts the new initiative to integrate traditional multidisciplinary debugging approaches and methodologies with multiplatform pattern-driven software problem solving, unified debugging patterns, best practices in memory dump analysis and software tracing, computer security, economics, and the new emerging trends I’m going to write about during this year.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -