Software Diagnostics Library

Sometimes we have several Threads of Activity (for example, from the same process) visible for a certain period of time and then suddenly we see only one such thread till the end of a trace (or even none). This may be an indication of an application hang or some other abnormal behaviour if several active threads doing logging are normal. If we consider such activities (including Adjoint Threads) as vectors running through some temporal “surface” we can use an analogy of a divergence and call this analysis pattern Activity Divergence:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we see Error Message or Exception Stack Trace in a log file we might want to explore the sequence of messages from the same Thread of Activity that led to the error. Such Message Set has an analogy with memory analysis patterns such as Execution Residue (of partial stack traces without overwrites) and Stack Trace (where the error message is a top stack frame) and we call this pattern Error Thread:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Most of the time, especially for large software logs, we need to select messages based on some criteria be it a set of Error Messages, a set of messages containing Basic Facts, or some other predicate. Then we can use selected messages from that Message Set as Anchor Messages or reverse Pivot Messages as an aid in further analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the powerful trace analysis techniques is using Adjoint Threads of Activity to filter various linear message activities (as a generalization of Thread Of Activity). Such filtered activities can then be analysed either separately (Sheaf of Activities) or together such as a new pattern we introduce here: Message Cover. If we identify parallel ATIDs (ATID is Adjoint TID, see an example) and see that one covers the other we can then make a hypothesis that they are correlated. Here is a graphical example of a Periodic Message Block largely composed from various Error Messages that covers periodic Discontinuities from another ATID (we can also consider the latter as periodic message blocks consisted from Silent Messages):

This is analogous to a cover in topology.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Introduced last year in Debugging TV Frames episode 0×32 about Android / Java debugging State Dump pattern solves the problem of a program state analysis when memory dump generation is not available or doesn’t help or complicated in the case of interpreted code. Basically a developer identifies a set of state variables and periodically prints their values to the output logging stream. Such output may also include but not limited to Counter Values.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

So far we have been discussing trace analysis patterns related to execution of a particular software version. However, software code changes and also its tracing and logging output: from large scale changes where components are replaced to small scale code refactoring affecting message structure and format. On a software narratological level this corresponds to a narrative about a software trace or log, it evolution. Such Meta Trace analysis pattern is different from Master Trace pattern where the latter is similar to what Metanarrative is usually meant in narratology: a master or grand idea - an expected trace if all functional requirements were correctly identified and implemented during software construction and non-functional ones are met during software execution.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we are interested in changes in particular {property, value} pairs or in tuples {x₁,x₂,x₃,…) in general where x_i can be a number or a substring. This is more general pattern than Message Change because such tuples can be from different sources and belong to different messages:

This pattern is also different from Data Flow where a value stays constant across different sources and messages. It is also different from Gossip pattern which involves more semantic changes. Metaphorically we can think of Data Association pattern as a partial derivative.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we know from Basic Facts some data or activity we seek to identify in different traces collected together to perform inter-correlational analysis. It can be a shared file name, a named synchronization object, a locked file with sharing violations, a common virtual address in kernel space, or just some activity notification. We call this pattern Shared Point by analogy with intersecting curves in some abstract space.

It is similar to Linked Messages pattern but is more high level and not confined to a common parameter (can be an action description).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We mostly analyse real messages in software traces and logs. In such message streams we may see easily detectable Discontinuity patterns. However, in some cases it is beneficial to analyse the absence of messages. Message stream is not uniform, there may be different currents. If time resolution is 1 ms, for example, then we may have a current N msg/ms or in the case of lesser current, such as, 0.5 msg/ms we have the so called Silent Messages (—-):

[...]
11 ms: message
12 ms: ----
13 ms: message
14 ms: ----
15 ms: message
16 ms: message
17 ms: ----
18 ms: ----
19 ms: message
[...]

So, by a silent message we understand the possible message that would occupy the minimal time resolution gap. If we look at the following illustration we would see that the whole pattern analysis apparatus can be applied to the analysis of distribution of silent messages.

Silent Messages pattern is different from Discontinuity pattern because the latter is about large unexpected silences and Sparse Trace which is about missing trace statements from source code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The next trace and log analysis pattern name is borrowed from Factor Groups in mathematics (or quotient groups). Here a group is, of course, not a mathematical group but just a group (or set) of log messages or trace statements. However, every trace message has variable and invariant parts. Variable parts usually contain some values, addresses or status bits. It can even be string values. Such values form a set too and can be partitioned into disjoint (non-overlapping) subsets. For example, a window foreground status can be either true or false. And we can group messages into disjoint factor groups each one having either only true or only false foreground status. The following trace graph illustrates a WindowHistory64 log where it was reported that one window was periodically loosing and gaining focus:

We found messages related to the reported process window title. By using Density Distribution pattern another such group of messages was found for another process window. Then a factor group was formed with two subgroups and their Relative Density was compared. For correlated alternating values it was expected to be 1. This was a very simple case, of course, which was analysed just by looking at a textual log but in more complex cases a computer assistance is required. A member of a factor group can also be generalized as a message subset with messages having variable part values from some domain subset or even calculated from it (some sort of a predicate): M_i = { m | P(m) }, where the original group of messages is a disjoin union of such message subsets: M = U M_i.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we find a grouping of some messages in one trace and then we are interested in the some groupings either in the same trace (Intra-Correlation) or in another trace (Inter-Correlation). We may consider such grouping as having some local density compared to global Message Density pattern. Then we might be interested in that selected message grouping Density Distribution illustrated on this minimal trace graph:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Activity Regions or blocks of messages having the same TID or PID usually follow each other in a typical complex software trace. Such following can be completely random and independent or it may be linear based on IPC or some inter-thread communication mechanism. For example, after filtering out Background Components we my find that an RPC client call setup is followed by messages from an RPC server:

Using a coordinate approach with message number and PID axes we can reformat this minimal trace diagram:

We call such pattern Piecewise Activity where we borrowed the concept of a piecewise linear function in mathematics (and piecewise continuity). In some problem software behaviour scenarios where we encountered such analysis pattern it was complemented by Discontinuity pattern. For example, an RPC call may be blocked and we don’t see client messages after that break till the end of the trace. In such cases we always recommended forcing a complete memory dump to check for wait chain memory analysis patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When analysing Inter-Correlation or Intra-Correlation and finding Discontinuities in one part or in a different trace (for example, in client-server environments) it is useful to see if there are corresponding Correlated Discontinuities in another part of the same trace (for example, a different Thread of Activity) or in a different trace. Such pattern may point to the underlying communication problem and may suggest to gather a different trace (for example, a network trace) for further analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Dialogue is an important pattern especially in network trace analysis. It usually involves a message source, a different message target (although both can be the same) and some alternation between them as shown on this abstract trace diagram:

Message and source are not only IP addresses or port numbers. They can be window handles, for example. Sometimes, the roles of source and target play different Process ID and Thread ID combinations (Client ID, CID). In such cases some parts of a message text may signify reply and response as shown graphically:

The similar illustration can be done for multi-computer trace, for example, when several traces from different servers are combined into one, where a combination of CID and a computer ID (Co) or just Co can play the roles of source and target.

Note that on all illustrations above the 3rd request doesn’t have a reply message: a possible Incomplete History pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we look at a trace or log and instead of Error Messages we only see their “signs” such as a DLL load event for an error or fault reporting module or a module that is related to symbol files such diasymreader.dll. We call this pattern Hidden Error by an analogy to Hidden Exception in memory dump analysis pattern catalogue although sometimes we can see such modules in memory dump Module Collection. For example, the presence of diasymreader module may signify an unreported .NET exception and suggest a dump collection strategy.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

For event- or message-driven architectures it is important to differentiate between event and state messages (including state transition). For example, a system may be doing some work while being in some particular state with a lot of tracing and respond to various external events with each of them having a corresponding trace message. Upon such an event the system transitions to some other state with its own set of possible trace messages. We call such a pattern State and Event. Typical example here is a windowing terminal services system and WM_ENDSESSION event illustrated in the following abstract trace diagram with a corresponding state transition diagram below it:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes in the case of missing Basic Facts we can discern Indirect Facts from message text and even from other patterns. For example, in one incident we were interested in all messages from the certain process name but its PID was missing from the problem description. Fortunately we were able to get its PID from one of individual messages from completely different source:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In the past we viewed software traces and logs as temporarily ordered event sequences. Since events are just memory data we have a map

T  -> M

as can be seen in the definition of a software trace. Here we generalize the domain to any arbitrary set, for example, it can be a list of indexes or pointers or even memory itself. The latter map can give us narrative chains such as

M -> M -> M -> M

and even give us a grand unification of memory and log analysis and the possibility to apply software narratology to memory dump analysis as well. We talk about it soon and provide some generalized software narrative examples.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

While preparing a presentation on malware narratives we found that one essential pattern is missing from the current log analysis pattern catalog. Most of the time we see some abnormal or unexpected value in a software trace or log such as a network address outside expected range and this triggers further investigation. The message structure may be the same having the same Message Invariant but variable part may contain such values as depicted graphically:

We call this pattern Abnormal Value and plan to add the similar one to memory dump analysis pattern catalog. Please not that we also have Significant Event pattern that is more general and also covers messages without variable part or just suspicious log entries.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -