Software Diagnostics Library

The dual analysis technique to Adjoint Spaces trace and log analysis pattern is called Memory Fibration (by analogy with fibrations in mathematics). Certain process spaces may have associated external logs so the dynamics of memory can be examined backwards. Such traces and logs may be the parts of just one bigger log as Adjoint Threads of Activity with their PIDs as ATIDs.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having chosen a trace message, we are interested in its Message Context which can span all “continuous” messages before and after from the same Thread of Activity. We it Activity Quantum which is variable and independent from the so-called CPU quantums. Different messages from Activity Quantum may be executed on different CPUs. The following diagram depicts this analysis pattern:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here we introduce Delay Dynamics analysis pattern. It is not an oxymoron and dynamics is referred to by what actually happens during the delay (Discontinuity with Time Delta) in other Threads of Activity as depicted in the following diagram:

Instead of threads various Adjoint Threads of Activity may also be inspected.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the trace attributes we didn’t pay much attention to in the past is CPU. This column is present in some ETW-based trace implementations such as Citrix CDF traces. As any trace attribute, it can be used to form Adjoint Thread of Activity (as all messages from code executed on that particular CPU). As we already considered threads as braids, we use braid groups as a further metaphor. In our case we combine CPUs and threads into one group which uses permutation for CPU scheduling. Instead of permutations, twists may be modeled as changes of threads. The Braid Group analysis pattern is illustrated in the following diagram:

This is a preliminary description of the analysis pattern. We plan to elaborate on it in further case studies. For example, instead of multithreading we can use multibraiding.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Recently we found a correlation between software trace with high Statement Density and Current of Periodic Error with uniform Error Distribution and process heap Memory Leak suspected from  memory dump analysis. If we metaphorically view periodic errors as “frequency” and the size of a heap as “mass” we may see that the growth of “frequency” correlates with the growth of “mass” and vice versa. Since frequency is inversely proportional to wave length we see a metaphorical analog to Louis de Broglie’s wave-particle duality. In general, as we already pointed in the discussion of narrativity and spatiality of software execution artifacts (see also Software Trace and Memory Dump Analysis seminar), software traces / logs and memory dumps can be seen as “dual” to each other according (metaphorically again) to de Broglie’s “duality of the laws of nature”. So we name this analysis pattern De Broglie Trace Duality since some memory dump regions can be considered of a general trace nature. Our correlation can be depicted in this diagram:

Practically, when we see Memory Consumption Patterns (but don’t know their source / root cause yet) we can ask for traces and logs, and in the case of frequent Periodic Errors found there we can suggest troubleshooting steps that may serve as a resolution or workaround.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Considering and extending Trace Dimension to TID, ATID, and Time we use Poincaré section and Poincaré map analogies to introduce Poincaré Trace analysis pattern. We choose a value of one “coordinate”, for example, Time, and then form the new trace that consists of messages from different (A)TIDs that have the same timestamp as Time value or the last message(s) from other different ATIDs if they happened before section Time:

If we choose a different ATID than Time and TID, then Poincare Trace is equivalent to Adjoint Thread of Activity. If ATID is TID we have Thread of Activity as a trivial case. If we have Thread of Activity, we can choose some ATID and get Poincaré Trace as illustrated in the following diagram where the thick black line in the right Poincaré Trace represents Discontinuity and its Time Delta:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There are messages in traces and logs that alone do not have useful information. These are adornments or ornamentation messages that we initially called Delineator or Separator messages that structure message stream or Figural Events (where we borrowed the name figural from Lyotard) such as formatting the next message with { }. Such messages only make meaning with other meaningful messages. We call this pattern Ornament. It helps in using trace analysis patterns and trace pattern identification.

Ornament can also be present inside individual messages, for example, as a character prefix. It can be a part of Message Invariant or variable itself. Some ornamentation can play a rhetorical function.

Although ornament is present in non-software architecture this concept has never been explored in pattern–oriented software architecture and construction. We propose similar concept for source code (software construction) in addition to ornamentation of its Declarative Trace. For software post-construction, we also add ornament part to software structure and behavior parts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we have hundreds of separate trace files from Split Trace and a smaller Split Trace with fewer files or just a single trace file that was recorded simultaneously (for example, a client from client-server environment) we can “project” the smaller Message Space into the larger Message Space as depicted in the following diagram:

This Projective Space analysis pattern can be used in conjunction with Indexical Trace where time interval can be used as an index into the larger Split Trace. Such projection may not be accurate but, assuming that the target trace Statement Current is uniform on average, can still be a very good heuristic instead of a binary search. For example, recently we had 4 sequential trace files for the client and 36 sequential files for the server. The software problem interval was specified in Basic Facts. We found that it in the second path of the 4th client trace. We, therefore, only inspected the last 4 traces of the 36 server sequential trace set and found it contained in the 35th server trace.

This pattern uses projective space metaphor from mathematics.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In Adjoint Message analysis pattern description we mentioned compressing message sequences having the same message attribute into one message. Considering the trace as “topological” space and message attribute as “equivalence” relation we introduce Quotient Trace analysis pattern by analogy with quotient space in topology. By endowing message sequences having the same attribute with some “metric” such as cardinality of Message Set we can also visually distinguish resulted quotient messages if they have the same attribute but from different sequences at different times. All this is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes log messages are formatted with mistakes, buffers are not cleared before copying, copied strings are truncated, tracing implementation and presentation contains coding defects. There can be internal corruption when messages are formed or “corruption” during presentation, for example, default field conversion rules (like in Excel). We call this pattern Corrupt Message. Such messages may affect trace and log analysis where data search may not show full relevant results. We then recommend to double check findings by using Data Flow of a different Message Invariant.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes, the presence of some messages in a trace or log shows that some other tracing or logging tool was running or that some process was also doing tracing. We call this analysis pattern Surveyor. Such discovered tracing may not be related to the trace we are looking at (compare to Trace Extension) but may help with finding additional traces in the system as illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Recently we performed the diagnostic analysis of a software incident where certain functionality was not available to users and provided the report based on analysis patterns such as Focus of Tracing and Opposition Messages. We also conjectured some hypotheses explaining the observed abnormal behaviour. However, at the end, the problem was solved not by the analysis of a lengthy software execution log but by looking at the small configuration INI file where not working functionality was simply disabled in one line:

EnableFunctionality = 0

Even before that analysis we were thinking about the importance of Small DA+TA such as configuration files and registry details that can be considered as general software traces. Here DA+TA means Dump Artefact + Trace Artefact and Big DA+TA refers to software execution memory dump artefacts and trace artefacts that can be really huge. The analysis pattern is illustrated in the following diagram where we see no difference between working and non-working scenarios due to insufficient trace coverage (Sparse Trace):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to Message Patterns there are higher level patterns of specific activities and Motives. Such activities may or may not coincide with specific components (modules) because they may be grouped based on implementation messages, software internals semantics and not on architectural and design entities (as in Use Case Trail analysis pattern). Moreover, he same components may “play” different activity roles. Once assigned, Activity Theatre “scripts” can be compared with “scripts” from other traces and logs (Inter-Correlation) or different parts of the same log (Intra-Correlation). This pattern is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Now we come to the trace and log analysis pattern that we call Message Pattern. It is an ordered set of messages from Thread of Activity or Adjoint Thread of Activity having Message Invariants that can be used for matching another ordered set of messages in another (Inter-Correlation) or the same trace or log (Intra-Correlation). A typical Message Pattern from one of our own trace and log analysis sessions is depicted in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Some tracing architectures, especially the ones that intercept API calls by filtering or hooking, may log synchronous requests by remembering to write done return result in the same trace message later on when the response is available after the wait. If such data is still not available in the log or trace it may point to some blocked request for which other software execution artifact analysis (such as memory dump analysis) is necessary. In some cases, the analysis of the corresponding Fiber Bundle stack trace may point to Blocking Module or the involvement of file system filters. This analysis pattern that we call Missing Data is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When using complex trace and log analysis patterns such as Fourier Activity we may be first interested in selecting all instances of a particular message type from specific Thread of Activity and then look for Time Deltas, Discontinuities, Data Flow, and other patterns. We call this analysis pattern Fiber of Activity by analogy of fibers (lightweight threads) since the individual thread execution flow is “co-operative” inside, whereas threads themselves are preempted outside. The following diagram from Fourier Analysis example illustrates the concept by showing three fibers:

This analysis pattern is different from trace-wide Sheaf of Activities where the latter is about selecting messages as Adjoint Threads of Activity which may span several processes and threads.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have trace and log messages that appear with certain time frequency throughout all log or specific Thread of Activity. Such frequencies may fluctuate reflecting varying system or process performance. Analyzing trace areas where such messages have different Time Deltas may point to additional diagnostic log messages useful for root cause analysis. The following minimal trace graph depicts the recent log analysis for proprietary file copy operation where the frequency of internal communication channel entry/exit Opposition Messages was decreasing from time to time. Such periods were correlating with increased time intervals between “entry” and “exit” messages. Analysis of messages between them revealed additional diagnostic statements missing in periods of higher frequency and corresponding Timeouts adding up to overall performance degradation and slowness of copy operation.

Additional analysis of Data Association in a different message type between available communication buffers and the total number of such buffers revealed significant frequency drop during constant Data Flow of zero available communication buffers:

We call this analysis pattern Fourier Activity by analogy with Fourier series in mathematics. This pattern is for individual message types and can also be considered a fine-grained example of Statement Current and Trace Acceleration analysis patterns which can be used to detect areas of different frequencies in individual Fibers (Adjoint Threads of Activities formed from the same Thread of Activity).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Extension is an obvious log analysis pattern that is about trace messages that refer to some other trace or log that may or may not exist. Sometimes, there can be instructions to enable additional tracing that is not possible to cover by the current trace source. We have seen this in some trace statements from .NET Exception Stack Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The trace statements in source code can be considered as Declarative Trace by analogy with variable declaration and definition in programming languages such as C and C++. Declaration of the variable doesn’t mean that the variable will be actually used. Some declared variables such as arrays will actually expand in memory when used (as in .bss sections). The same is with trace messages from Declarative Trace. Some of them will not appear in the actual software execution trace and some will be repeated because of loops and multiple code reentrance. However, Declarative Traces are useful for studying the possibilities of tracing and logging design, implementation, and coverage (for example, Sparse Trace). Some trace analysis patterns are also applicable for Declarative Traces such as Message Sets and Bifurcation Points (among different source code versions). This is illustrated in the following picture:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Variable Subtrace analysis pattern was introduced for inter-correlational analysis of CPU spikes across memory snapshots with just one thread involved. In contrast, we found Constant Subtrace pattern useful in Wait Chain analysis involving several threads in just one memory snapshot (intra-correlational analysis). Here a constant subtrace groups stack traces from Stack Trace Collection with a bifurcation stack trace frame (similar to Bifurcation Point trace analysis pattern) providing some wait chain relationship hint. Such subtraces traces may be initially found by the preceding wait chain analysis or by technology-specific subtraces such as ALPC/RPC server thread frames (as seen in an example stack from COM interface invocation). Here is a minimal stack trace diagram (similar to minimal trace graphs introduced in Accelerated Windows Software Trace Analysis training) illustrating the pattern (it also shows Spiking Thread pattern in user space as seen from a complete memory dump):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -