Sometimes in the case of missing Basic Facts we can discern Indirect Facts from message text and even from other patterns. For example, in one incident we were interested in all messages from the certain process name but its PID was missing from the problem description. Fortunately we were able to get its PID from one of individual messages from completely different source:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In the past we viewed software traces and logs as temporarily ordered event sequences. Since events are just memory data we have a map
T -> M
as can be seen in the definition of a software trace. Here we generalize the domain to any arbitrary set, for example, it can be a list of indexes or pointers or even memory itself. The latter map can give us narrative chains such as
M -> M -> M -> M
and even give us a grand unification of memory and log analysis and the possibility to apply software narratology to memory dump analysis as well. We talk about it soon and provide some generalized software narrative examples.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
While preparing a presentation on malware narratives we found that one essential pattern is missing from the current log analysis pattern catalog. Most of the time we see some abnormal or unexpected value in a software trace or log such as a network address outside expected range and this triggers further investigation. The message structure may be the same having the same Message Invariant but variable part may contain such values as depicted graphically:
We call this pattern Abnormal Value and plan to add the similar one to memory dump analysis pattern catalog. Please not that we also have Significant Event pattern that is more general and also covers messages without variable part or just suspicious log entries.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we have insufficiently detailed problem description and/or there are several similar parallel user activities going on simultaneously, for example, several session are launched in a terminal services environment. In such cases when tracing is done for the duration of specific user activity Dominant Event Sequence pattern may help. Here we select the most full sequence of events or event sequence based on some Basic Facts. For example, if session id was missing in problem description we can choose the most long and full process launch seqence and assume that its session id was the one missing:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Suppose we form an adjoint thread based on some message or operation type or some other attribute:
But we do not know where to start to look backwards for any anomalies relevant to our problem:
We go back to our full trace and find a problem message:
Although it is not in our adjoint thread we formed previously it is still considered as Pivot Message helping us to go backwards there:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
If Implementation Discourse focuses on objective technology specific discourse then Traces of Individuality pattern focuses on subjective elements in a software log and its messages. Here we mean some specific naming or tracing conventions either from an individual engineer habit or from some corporate coding standard. As an example of it consider a trace message from a catch statement:
"Surprise, surprise, should have never been caught"
More examples for this pattern will be added later.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Inter-Correlation analysis between a normal and a problem logs to find a Bifurcation Point (and a possible root cause) becomes a difficult task when both traces come from different environments with widely differing Background Components. Here a new analysis pattern called Sheaf of Activities (borrowed from sheaves from mathematics) can help. Basically this pattern is also a tool in tracking properties of trace message subsets. First we find out important message types around some Activity Region where we hope to find a difference between two traces:
Then we create several Adjoint Threads from different message types, for example, based on operation type or function name:
Then we analyze subtraces separately to find out a bifurcation point in each of them and then use this knowledge to find out differences between the original full traces.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Indexical Trace pattern describes an Inter-Correlation pattern variant when we have a trace that has messages of interest pointing to specific activity regions in another trace. The latter trace can be very huge, from another computer and split into many parts (Split Trace). This pattern is very helpful when the problem needs to be diagnosed in the large split trace but we don’t know when it happened. Then an index trace that may have recorded software execution account (for example, in the case of a broker-like architecture) and can point to the right trace fragment from the split trace.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Static program analysis is used to eliminate certain coding errors that may lead to abnormal software behaviour. So it is naturally a part of software diagnostics but at source code level. Our goal here is to identify certain patterns directly linkable to patterns we see in memory dumps and software logs and collect them into a catalog. One such pattern candidate is called Loop Construct. It covers conditional and unconditional loops, for example, in one of modern languages:
extern bool soonToBeTrue; int mediumValue = ...;while (true){ TRACE("Waiting"); sleep(mediumValue); if (soonToBeTrue) { break; } doHeavyWork(); }
while (--pControl->aFewPasses)
{
TRACE("Waiting");
sleep(mediumValue);
doHeavyWork();
}Such loops may potentially lead to Spiking Thread memory dump analysis and High Message Current and Density trace analysis patterns. Of course, we shouldn’t suspect every loop but only some that have potential to be altered by Local Buffer Overflow (for mediumValue) or Shared Buffer Overwrite (for Control.aFewPasses) or by a race condition (soonToBeTrue).
We expect things to get more interesting when we start associating source code that uses certain API with patterns of abnormal behavior.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
If you need to get various real life software traces with millions of lines from complex software environments to learn pattern-driven software log analysis you can us Process Monitor as a modeling tool. Here you can abstract from their “monitoring” and API interception context and consider trace messages as emitted from various processes and threads (like Citrix CDF traces). This approach was used in Accelerated Windows Software Trace Analysis training and Debugging TV Frames episode 0×19.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This pattern is called Opposition Messages (borrowed from binary opposition originated in Saussure’s structuralism). It covers the following pairs of messages usually found in software traces and logs such as:
The absence of an opposite may point to some problems such as synchronization and leaks or Incomplete History (wait chains). There can always be a possibility that a second term is missing due to Sparse Trace but this is a poor implementation choice that leads to confusion during troubleshooting and debugging.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Some tracing tools such as Citrix CDFControl have an option to split software traces and logs into several files during long recording. Although this should be done judiciously it is really necessary sometimes. We call this pattern Split Trace. What to do if we get several trace files and we want to use some other analysis tool such as Citrix CDFAnalyzer? If we know that the problem happened just before the tracing was stopped we can look at the last few such files from the file sequence (although we recommend a circular trace here). Otherwise we can convert them into CVS files and import into Excel which also supports adjoint threading.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
To celebrate the last unique occurrence of 12.12.12 in this Century I created a Computical Art (Computicart) picture that uses minimalist software trace analysis diagrams I use for trace and log analysis patterns made even better for Accelerated Windows Software Trace Analysis course. The following diagram symbolizes several patterns. Suppose we trace personal or world events each year and form an Adjoint Thread with messages having DD=MM=YY timestamp invariant (or filter them). Then we clearly see a Discontinuity before the next Century with much bigger Time Delta than between such messages. Also these messages form a Periodic Message Block in relation to the full trace.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We have decided to factor out Anchor Messages example of message interleave into another pattern called Message Interleave. It covers superposition of different anchor messages, for example, process launch and exit, or DLL load and unload:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we don’t see anything in the trace or see very little because particular source code fragment was not covered by trace statements (see also PLOTs):
We call this pattern Sparse Trace and this is different from Missing Component pattern where some modules were not included for tracing explicitly although there is tracing code there or Visibility Limit pattern where tracing is intrinsically impossible. Often technical support and escalation engineers request to add more trace statements and software engineers extend tracing coverage iteratively as needed.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I get many questions on whether software log analysis patterns from Software Diagnostics Institute are OS or platform or product specific. My answer is that they are independent from all of them because they are based on viewing software logs as stories of computation and were discovered by application of narratological analysis (software narratology). In addition to these patterns there exist domain specific problem patterns such as wrong hotfix level or specific product error code during software installation or execution. Typical examples of support for such platform and product specific type of patterns include Microsoft Windows Problem Reporting and Citrix Auto Support.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Most of the time it is not possible to trace from the beginning. Obviously, internal application tracing cannot trace anything before that application start and its early initialization. The same is for system wide tracing which cannot trace before the tracing subsystem or service starts. Therefore, each log has its Visibility Limit in addition to possible truncation or missing components:
One of solutions would be to use different tracing tools and inter-correlation to glue activities, for example, Process Monitor and CDFControl.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Doing memory dump analysis training for more than 2 years I found that students are divided into 2 types: those who prefer to see source code first and those who want to see a memory dump first. We actually prefer to show a memory dump first and then explore it to find certain patterns of abnormal structure and behavior. Software Diagnostics Services used this approach to design its Accelerated Windows Memory Dump Analysis and Accelerated .NET Memory Dump Analysis courses. Students explore memory dumps and debugger logs to find memory dump analysis patterns which are introduced when necessary. After that they can check source code of modeling applications if they have development experience. Accelerated Windows Software Trace Analysis course uses a different approach. It introduces all software trace analysis patterns at once because they are patterns from software narratology independent from programming languages and software platforms. After that they explore and analyze software traces and logs. We can summarize these 2 approaches on this diagram:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Relative Density pattern describes anomalies related to the semantically related pairs of trace messages, for example, “data arrival” and “data display”. Their statement densities can be put in a ratio (also called specific gravity) and compared between working and non-working scenarios. Because the total number of trace messages cancel each other we have just mutual ratio of two message types. In our hypothetical “data” example the increased ratio of “data arrival” to “data display” messages accounts for reported visual data loss and sluggish GUI.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
If trace messages contain some character or formatted data that is passed from module to module or between threads and processes it is possible to trace that data and form a Data Flow thread similar to an adjoint thread we have when we filter by a specific message. However, in the former case we have different message types.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -