Trace and log analysis starts with assessment of artifact File Size, especially with multiple logging scenarios in distributed systems. If all log files are of the same size we might have either Circular Traces or Truncated Traces. Both point to wrong trace timing plan or just using default tracing tool configuration.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Trace messages may correspond to specific implementation code such as recording the status of an operation, dumping data values, printing errors, or they may correspond to higher levels of software design and architecture, and even to use case stories. We call such messages Milestones by analogy with project management. Alternative names can be Chapter Messages, Summary Messages, Checkpoints, or Use Case Messages. These are different from Macrofunctions which are collections messages grouped by some higher function. Milestone messages are specifically designed distinct trace statements:
They can also be a part of Significant Events, serve the role of Anchor Messages, and be a part of Basic Facts and Vocabulary Index.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we have several use case instances traced in one log file. Messages and Activity Regions from many Use Case Trails intermingle and make analysis difficult especially with the absence of UCID (Use Case ID), any other identification tags, or Message Links. However, initially most of the time we are interested in a sequence of Significant Events. After finding Anchor Messages we can use Time Deltas to differentiate between trace statements from different Use Case Trails. Here we assume correct Event Sequence Order. We call this pattern Event Sequence Phase by analogy with wave phases (all such individual “waves” may have different “shapes” due to various delays between different stages of their use case and implementation narratives):
In the picture above we also identified Dominant Event Sequence for use case instance C.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Use cases are implemented in various components such as subsystems, processes, modules, and source code files. Most of the time with good tracing implementation we are able to see Use Case Trails: log messages corresponding to use case scenarios. For simple systems one log may fully correspond to just one use case, but for complex systems, especially distributed client-server ones, there may be several use case instances present simultaneously in one log. One way to disentangle them in the absence of UCID (Use Case ID) or some other grouping tag is to use Event Sequence Phase, the subject of the next pattern.
Master Traces may also correspond to use cases but they should ideally correspond to only one use case instance.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes the absence of messages, for example, errors and exceptions, may save time during troubleshooting and debugging by pointing to what wasn’t happening and provide additional insight. For example, on the picture below we see the same exceptions in the new and old incidents. However, in the old incident we see another exception that was linked to one unavailable server in a distributed broker architecture. Therefore, we can assume provisionally that all servers were operational when the new incident happened.
Missing Message pattern is different from Missing Component pattern where the latter may point to the component that was not loaded or executed, or simply that it wasn’t selected for tracing.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We recently analysed a Process Monitor log which had a several hour gap. So we decided to describe it as Blackout pattern. If you see such a pattern it might have the following possible causes:
- Some files from Split Trace are missing
- Split Trace file set was artificially created
- The tracing scope system was paused or frozen (for example, a virtualized system), or restarted
- The tracing itself was paused
Blackout is different from Visibility Limit where the latter is about the inherent inability to trace whereas the former is only temporary inability due to circumstances listed above. It is also different from Discontinuity pattern where the latter is about gaps in individual Threads of Activity or Adjoint Threads of Activity
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Usually when we analyse traces and find an Anchor or Error Message we do backtracking using a combination of Data Flow and Message Sets and selecting appropriate log messages to form Back Trace leading to a possible root cause message:
This pattern is different from Error Thread pattern which just backtracks messages having the same TID (or in general ATID). It is also different from Exception Stack Trace pattern which is just a serialized stack trace from memory snapshot.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The previous patterns such as Basic Facts and Vocabulary Index address the mapping of a problem description to software execution artefacts such traces and logs. Indirect Facts analysis pattern addresses the problem of an incomplete problem description. However, we need another pattern for completeness that addresses the mapping from a log to troubleshooting and debugging recommendations. We call it Hidden Facts which are uncovered by trace analysis. Of course, there can be many such hidden facts and usually they are uncovered after narrowing down analysis to particular Threads of Activity, Adjoint Threads, Message Context, Message Set, or Data Flow. The need for that pattern had arisen during the pattern-oriented analysis of the trace case study from Malcolm McCaffery and can be illustrated on this diagram:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we have several Threads of Activity (for example, from the same process) visible for a certain period of time and then suddenly we see only one such thread till the end of a trace (or even none). This may be an indication of an application hang or some other abnormal behaviour if several active threads doing logging are normal. If we consider such activities (including Adjoint Threads) as vectors running through some temporal “surface” we can use an analogy of a divergence and call this analysis pattern Activity Divergence:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
When we see Error Message or Exception Stack Trace in a log file we might want to explore the sequence of messages from the same Thread of Activity that led to the error. Such Message Set has an analogy with memory analysis patterns such as Execution Residue (of partial stack traces without overwrites) and Stack Trace (where the error message is a top stack frame) and we call this pattern Error Thread:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Most of the time, especially for large software logs, we need to select messages based on some criteria be it a set of Error Messages, a set of messages containing Basic Facts, or some other predicate. Then we can use selected messages from that Message Set as Anchor Messages or reverse Pivot Messages as an aid in further analysis.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
One of the powerful trace analysis techniques is using Adjoint Threads of Activity to filter various linear message activities (as a generalization of Thread Of Activity). Such filtered activities can then be analysed either separately (Sheaf of Activities) or together such as a new pattern we introduce here: Message Cover. If we identify parallel ATIDs (ATID is Adjoint TID, see an example) and see that one covers the other we can then make a hypothesis that they are correlated. Here is a graphical example of a Periodic Message Block largely composed from various Error Messages that covers periodic Discontinuities from another ATID (we can also consider the latter as periodic message blocks consisted from Silent Messages):
This is analogous to a cover in topology.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Introduced last year in Debugging TV Frames episode 0×32 about Android / Java debugging State Dump pattern solves the problem of a program state analysis when memory dump generation is not available or doesn’t help or complicated in the case of interpreted code. Basically a developer identifies a set of state variables and periodically prints their values to the output logging stream. Such output may also include but not limited to Counter Values.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
So far we have been discussing trace analysis patterns related to execution of a particular software version. However, software code changes and also its tracing and logging output: from large scale changes where components are replaced to small scale code refactoring affecting message structure and format. On a software narratological level this corresponds to a narrative about a software trace or log, it evolution. Such Meta Trace analysis pattern is different from Master Trace pattern where the latter is similar to what Metanarrative is usually meant in narratology: a master or grand idea - an expected trace if all functional requirements were correctly identified and implemented during software construction and non-functional ones are met during software execution.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we are interested in changes in particular {property, value} pairs or in tuples {x1,x2,x3,…) in general where xi can be a number or a substring. This is more general pattern than Message Change because such tuples can be from different sources and belong to different messages:
This pattern is also different from Data Flow where a value stays constant across different sources and messages. It is also different from Gossip pattern which involves more semantic changes. Metaphorically we can think of Data Association pattern as a partial derivative.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we know from Basic Facts some data or activity we seek to identify in different traces collected together to perform inter-correlational analysis. It can be a shared file name, a named synchronization object, a locked file with sharing violations, a common virtual address in kernel space, or just some activity notification. We call this pattern Shared Point by analogy with intersecting curves in some abstract space.
It is similar to Linked Messages pattern but is more high level and not confined to a common parameter (can be an action description).
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We mostly analyse real messages in software traces and logs. In such message streams we may see easily detectable Discontinuity patterns. However, in some cases it is beneficial to analyse the absence of messages. Message stream is not uniform, there may be different currents. If time resolution is 1 ms, for example, then we may have a current N msg/ms or in the case of lesser current, such as, 0.5 msg/ms we have the so called Silent Messages (—-):
[...]
11 ms: message
12 ms: ----
13 ms: message
14 ms: ----
15 ms: message
16 ms: message
17 ms: ----
18 ms: ----
19 ms: message
[...]
So, by a silent message we understand the possible message that would occupy the minimal time resolution gap. If we look at the following illustration we would see that the whole pattern analysis apparatus can be applied to the analysis of distribution of silent messages.
Silent Messages pattern is different from Discontinuity pattern because the latter is about large unexpected silences and Sparse Trace which is about missing trace statements from source code.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The next trace and log analysis pattern name is borrowed from Factor Groups in mathematics (or quotient groups). Here a group is, of course, not a mathematical group but just a group (or set) of log messages or trace statements. However, every trace message has variable and invariant parts. Variable parts usually contain some values, addresses or status bits. It can even be string values. Such values form a set too and can be partitioned into disjoint (non-overlapping) subsets. For example, a window foreground status can be either true or false. And we can group messages into disjoint factor groups each one having either only true or only false foreground status. The following trace graph illustrates a WindowHistory64 log where it was reported that one window was periodically loosing and gaining focus:
We found messages related to the reported process window title. By using Density Distribution pattern another such group of messages was found for another process window. Then a factor group was formed with two subgroups and their Relative Density was compared. For correlated alternating values it was expected to be 1. This was a very simple case, of course, which was analysed just by looking at a textual log but in more complex cases a computer assistance is required. A member of a factor group can also be generalized as a message subset with messages having variable part values from some domain subset or even calculated from it (some sort of a predicate): Mi = { m | P(m) }, where the original group of messages is a disjoin union of such message subsets: M = U Mi.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we find a grouping of some messages in one trace and then we are interested in the some groupings either in the same trace (Intra-Correlation) or in another trace (Inter-Correlation). We may consider such grouping as having some local density compared to global Message Density pattern. Then we might be interested in that selected message grouping Density Distribution illustrated on this minimal trace graph:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Activity Regions or blocks of messages having the same TID or PID usually follow each other in a typical complex software trace. Such following can be completely random and independent or it may be linear based on IPC or some inter-thread communication mechanism. For example, after filtering out Background Components we my find that an RPC client call setup is followed by messages from an RPC server:
Using a coordinate approach with message number and PID axes we can reformat this minimal trace diagram:
We call such pattern Piecewise Activity where we borrowed the concept of a piecewise linear function in mathematics (and piecewise continuity). In some problem software behaviour scenarios where we encountered such analysis pattern it was complemented by Discontinuity pattern. For example, an RPC call may be blocked and we don’t see client messages after that break till the end of the trace. In such cases we always recommended forcing a complete memory dump to check for wait chain memory analysis patterns.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -