Software Diagnostics Library

Sometimes, we ask for a log file to see State and Event pattern, and see it there, only to find that we cannot do Back Trace of State Dumps from some Significant Event for Inter-Correlation analysis because our Data Interval is truncated (Truncated Trace). This highlights the importance of proper tracing intervals that we call Significant Interval analysis pattern by analogy with significant digits in scientific measurements. The following diagram illustrate the pattern:

If you find out you get truncated traces and logs often you may want to increase Statement Current for state logging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can “integrate” trace message stream into another, smaller trace. By analogy with motivic integration in contemporary mathematics we call this analysis pattern Motivic Trace. There can be border cases where the whole trace is reduced to one message or every message is associated with a different message (perhaps shorter or a number). Message Sets that are integrated into Motivic Trace can be completely different (for example, based on Motives) in comparison with Quotient Trace where we reduce Message Sets that have the same common attribute.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Using the metaphor of renormalization from physics we introduce Renormalization trace and log analysis pattern where a selected message and its Message Context are replaced by a single message:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

According to the definition in “Topological Signal Processing” by Michael Robinson (ISBN: 978-3662522844) “a signal consists of a collection of related measurements” (p. 5). For traces and logs we can apply the similar definition and consider Signal as a collection of local messages having the same Message Invariant and related variable data values. Signals are examples of Message Sets. The typical example are sets of related Counter Value messages. Signals can be obtained by obtaining Adjoint Thread of Activity of a specific message (to filter out Background Components “noise”) as illustrated in the following diagram:

Generally, the variable “measurement” part can form Braid of Activity.

We introduce Signal analysis pattern to bridge the gap between Software Narratology and Hardware Narratology.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We have analysis patterns that compare changes in software traces and logs during different executions (Master Trace) and during the evolution of software itself (Meta Trace). Such patterns are general enough, and often we are interested in their restriction to different execution paths or changes in code that leave start and end software states invariant:

We call such analysis pattern Trace Homotopy by analogy with homotopy in mathematics where a curve or sequence of operations can vary with constant endpoints.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we analyze traces and logs we work with only a small subset of log messages. We call any such current subset Working Set by analogy with working sets in operating system memory paging implementations:

This analysis pattern can also be reconciled with an operadic approach to trace and log analysis by chaining appropriate diagnostic operads from the original traces to the desired working sets:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Messages that contain scripting statements can be signs of malnarratives that resulted from log injection during attempts to exploit possible cross channel scripting (XCS) and cross-site scripting (XSS) vulnerabilities. Such Script Messages may be spread across a log as illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can associate a function with a domain of trace messages (M) to some other range either continuous (T) or discreet (D). We call this analysis pattern Trace Field by analogy with fields in physics:

Or, in general, Trace Field, is a functor between the domain of the category of trace messages (M) and a codomain of some other category, not necessarily numerical. Typical examples include Trace Presheaves and Fiber Bundles. Another example, for generalized logs, is Memory Fibration taken to extreme.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have variable sequences of Significant Events when we expect the certain constant number of such events in repeated normal and problem cases. For example, in repeated normal cases we expect more than 10 of events but in repeated abnormal cases - just 2 events. The latter cases would indicate that something happened inside the processing of the second event. But if one or more such cases contain 3 or 4 events that would point to some external influence that aborted the sequence of events. We call such variable event sequences Hedges by analogy with hedge variables or variadic variables. One of the recent example that we encountered involved multiple abnormal cases with just 2 events. We were about to investigate the internals of the second event but noticed that one of the cases contained 3 events. Further analysis indicated that the whole sequence was aborted by some external process after reaching the certain timeout. In the case of 3 events the first 2 happened too earlier and that allowed for the 3rd event to happen before the timeout was triggered:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Certain types of blind SQL injection attacks may leave log messages with just one byte difference. We call with analysis pattern Ultrasimilar Messages by analogy with an ultrametric space in mathematics and the interpretation of messages as p-adic numbers. Since, such messages may be scattered in a log we can choose Message Pattern based on some Message Invariant (for example, parts of SQL request) and then analyze its Fiber of Activity (for example, Data Flow of its variable part). A log with two different types of Ultrasimilar Messages is shown in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we are interested in Message Set that has the same time attribute value (or rounded to some digit). We call this analysis pattern Activity Packet by analogy with wave packets. It may allow identification of related threads and activities.

It is different from Activity Quantum analysis pattern where time attribute value may change for continuous Message Set with the same thread id.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A typical software trace or log (for example, from Process Monitor) lists messages from several processes and threads sequentially. However, such columns may be split into individual process ID or thread ID columns. The same can be done for any Adjoint Thread and illustrated in the following diagram:

We call this analysis pattern Combed Trace by analogy with multibraiding.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In order to perform Inter-Correlational analysis among traces and logs especially when we have Indexical Trace we need a dual operation: an ability to identify traces and Use Case Trails, and, if necessary, find their corresponding Braids of Activity in an index trace. Some data from the tracing domain or use case description may serve is Intrinsic ID. It can be itself some numeric ID, user or computer name.

A typical log analysis case from a distributed environment is illustrated on the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Often, for Inter-Correlational trace and log analysis, we need to make sure that we have synchronized traces. The one version of Unsynchronized Traces analysis pattern is depicted in the following diagram where one trace ends  (possibly Truncated Trace) before the start of another trace and both were traced within one hour:

If tracing was done in different time zones with different local times specified in logs we can determine whether the traces are synchronized (when time zone information is not available in Basic Facts) by looking at minutes as shown for the third trace in the diagram above. This technique can also be used in trace calibration (see Calibrating Trace).

There is a similar analysis pattern for memory analysis called Unsynchronized Dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The idea of Tensor Trace analysis pattern initially appeared in the context of memory dumps as general traces with several special traces inside but then developed further when working on Singleton Trace analysis pattern when we realized that several Singleton Traces may form a new separate log:

Therefore, we may combine several traces and logs into one global trace where each message references separate local traces and logs:

Typical example is a repeated tracing. Each trace has an i-th index spanning the number of trace messages. We say it is has T_i components. Each individual logging has an j-th index and overall global log has T^j components. Together the form the second rank tensor:

There can be Tensor Traces of the higher rank, for example, the 3rd component spanning computers:

This analysis pattern is different from Meta Trace where the latter is about trace evolution during software development. It is also different from Trace Dimension which is about one trace (Tensor Trace of rank 1).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If we consider a log as a text, ignore its column structure, and do search for the particular attribute value (for example, PID) we get Message Set consisting of messages having that attribute value as column (Adjoint Thread of Activity) and messages having that attribute value referenced in their message text. We call this pattern Braid of Activity because metaphorically it looks like Adjoint Threads of Activity cross each other (like multibraiding):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Status updates, error message boxes, and even abort errors can all be considered as examples of software traces and logs. We call such a log with one message only Singleton Trace. Please don’t confuse it with Singleton Event analysis pattern which is trivial for Singleton Traces. We illustrate this pattern with this picture of error message (in Russian it means “The machine doesn’t work. Code: SB”) from a lemonade-dispensing machine:

The message in such a trace may contain the associated stack trace as a trivial example of Fiber Bundle.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When doing Inter-Correlational analysis between different traces and logs we can move simultaneously along time direction or backwards (Back Tracing). In some cases we may start our analysis with the identification of Significant Events in both logs and then move in opposite directions finding another pair of messages that can be useful for diagnostic identification. We call this analysis pattern Galois Trace by analogy with Galois connection in mathematics. In our case moving from trace A to trace B and back corresponds to F and G functions with a and b as individual trace messages or their Message Contexts (as per monotone definition). The analysis pattern is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can apply Message Set trace and log analysis pattern to itself and build multilevel Message Directory. The pattern name come from the analogy with memory management page tables, where we can select every 10th message to build a smaller Message Set, and then select every 10th message from it to build another Message Set, and so on. When looking at a higher level Message Set we can select a trace message and then look at its Message Context in the lower level Message Sets:

Message Directory is similar to Trace Frames analysis pattern but is fine-grained, and more individual message oriented than the latter analysis pattern. However, the former pattern can be used to implement the latter one.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Adjoint Space and Trace Fibration analysis patterns may be useful in cases where complementing traces with memory dumps and vice versa provides better insight into software behavior. Ideally, every trace statement should have Adjoint Space but this is not feasible practically. The solution is to save memory regions surrounding trace message data, for example, structures referenced by pointers. This can be done either for each message or for selected Message Sets. Such memory data can be embedded inside logs as State Dump, Trace Extension or Inter-Correlation with a binary log of such memory fragments. This looks like a mapping between trace messages and memory objects. We call this analysis pattern Trace Presheaf by analogy with presheaves in mathematics. In case Adjoint Spaces are available (for example, memory dumps) such memory buffers can be written to memory by a debugger (in Windows by .readmem WinDbg command) and examined in association with the rest of Adjoint Space. This is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -