Software Diagnostics Library

Message Complex takes inspiration from simplicial complexes. We select a message, choose TID or ATID, and connect to the nearest messages having the same TID or ATID. This procedure can be repeated for newly connected messages. Then we select another ATID and repeat the procedure. Three connected messages with the same ATID may form a triangle and may also intersect another triangle with a different ATID if they share the same message. A very simple example is illustrated in the following diagram:

Message Complex is more structural and geometric compared to Message Context, which is just a set of surrounding messages regardless of their TID or ATID based on some relationship criteria.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

(Adjoint) Threads of Activity can be interpreted as braids (multibraiding). This braid analogy assumes that all (adjoint) threads implicitly start and end outside of the trace boundaries. However, some (adjoint) threads may start after the beginning of the tracing or end before the finishing of the tracing. Such modified braids are called braidoids. There can be several braidoids per trace based on the chosen (A)TIDs. We call this analysis pattern Trace Braidoids and one, based on TID, is illustrated in the following diagram:

We added arc crossings when a different TID becomes current. Please also compare these crossings with other analysis patterns such as Braid Group and Braid of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Message data may point to other messages in the same trace (see the example of Linked Messages) or in the other trace (see the Data Selector example). But similar data in other messages may not point to any other messages in the same or other, perhaps Truncated, traces and logs collected at the same time - similar to invalid pointers, for example, kernel addresses in process memory dumps or user space addresses in kernel memory dumps. We call this analysis pattern Null Reference (also notice the analogy with foreign key values in data tables where Null is not a value). Another analogy here is referential failure. This is illustrated in the following diagram adapted from Linked Messages analysis pattern diagram.

This analysis pattern is different from Missing Data where the reference is itself missing.

These Null References can be remediated by longer supplemental traces, Fiber Bundle, and Adjoint Spaces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we disregard the length of message blocks having the same attribute (ATID), we get Quotient Trace. But when we disregard the content of these message blocks (and replace each message with the same “empty” non-Silent Message) but preserve their length, we get Trace Skeleton.

Different Trace Shapes may have different Trace Skeletons but we can generate similar shapes from one skeleton.

We can also apply a music metaphor and consider it as Trace Rhythm:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we want to collapse messages into one message while preserving content, for example, for grep. We call such an analysis pattern Collapsed Message (by an analogy for collapsing an internal directed graph edge, or edge contraction). Several consecutive messages having the same attribute, for example Thread of Activity for Exception Stack Trace may be collapsed into one longer trace message. This is a simple case of Quotient Trace without compression or transform. And this is different than Motivic Trace which doesn’t preserve message content.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Various types of Measurements are important in software diagnostics. We consider traces and logs, and general software narratives (and even hardware narratives) as a medium for all types of possible measurements. Even a small display in a handheld device showing a number is an example of Singleton Trace.

Typical trace and log measurement analysis patterns include Time Delta, Statement Density and Current, and Trace Acceleration. Numeric analysis patterns include Counter Value, Trace Field, Signal in general, and the forthcoming Trace Distance that uses various metrics, for example, the number of messages, Activity Regions, or just hops.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Even if traces and logs are perfectly synchronized (Unsynchronized Traces) we me still get Unsynchronized Messages. If respective message times are the same (can depend on time resolution) we don’t know which one was first. This is visible if we do Trace Mask using different order:

If both messages belong to the same Thread of Activity we may be able to reorder them correctly based on additional message semantics, such as module hierarchy (for example, OS runtime library and application code that are traced separately).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Usually, when we find an interesting message in a log (maybe also a frame from Exception Stack Trace), especially from an unfamiliar component, we also want to search past problem cases either on the Internet or in some internal database. However, we just put the message as is or some small fragment of it we may get a lot of noise results. The problem is to find the optimal Message Essence. Often, this is done by omitting variable data (including Adjoint Thread of Activity fields) but leaving Message Invariants and Trace Constants usually refine a diagnostic error:

This analysis pattern is different from Message Invariant. where the latter is useful when finding its emitter’s source code lines (PLOT).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we have different traces and logs not necessarily with the same Trace Schema and select only messages that have some condition, for example, the same ATID (see Adjoint Thread of Activity) or FID (see Feature of Activity) value, we get the new trace that we call Trace Join. A combination of ATID from one trace or Message Set from another is also possible as illustrated in this allegorical picture when joining is done by “Plato” author value or title containing “Plato” (all case-insensitive):

This is very similar to relational data joins. Join of the same trace is possible too. A Dia|gram picture (similar to the previous patterns) is left as an exercise.

We initially wanted to call this analysis pattern Filtered Mask but later realized that it may not be possible to do Trace Mask if there is no global ordering information, such as time. In such a case, Serial Trace is possible. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The name of this pattern, Container Trace, originates from the logging style that is recommended as the best practice for containers (like Docker), where various components output their tracing and logging statements to the standard console output. Such components may have their own incompatible Trace Schemas, for example, normal trace messages intermingled with Exception Stack Traces. However, in general, this pattern can be extended to any log file (a container for trace statements). This pattern is different from Trace Mask, where individual traces come from separate files and have Trace Schema with some ATID (see Adjoint Thread of Activity) or FID (see Feature of Activity), such as time, that allows for blending them correctly. Components that output their messages to Container Trace may not even have any internal Trace Schema. In such a case, Container Trace may simply be treated as Text Trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Most of the time tracing and logging is done sequentially, for example, when a service or application is restarted after the crash or bug fix, or the host is rebooted. Then we can glue all resulted traces together (similar to Glued Stack Trace) into one large Serial Trace. Here we assume the same Trace Schema for all individual traces and logs. It can also be considered as flattening a 2-dimensional Trace Tensor:

This allows us to apply various trace and log analysis patterns to the unified Serial Trace instead of doing Inter-Correlation (vs. Intra-Correlation).

Serial Trace is different from Meta Trace which is a trace about trace and Master Trace which is a trace we compare all other traces to. It is similar to Trace Mask when there is no overlap in time. Also, Serial Trace is not a reverse of Split Trace in a general case due to Visibility Limits between individual traces.

When gluing traces together, Ornament messages may be added to serve as a boundary between fragments.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Schema can be represented as Schema Trace or, avoiding naming confusion, Definition Trace. The resulting trace looses ordering (similar to unordered Message Set) but allows application of trace and log analysis patterns, especially if some order is fixed, for example, alphabetical for names or original presentation column arrangement. Schema definition Trace Schema can be represented as another Definition Trace as illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Most of trace and log analysis pattern illustrations using Dia|gram language are of these two general forms:

Although the first form represents typical ETW trace attributes, the analysis pattern descriptions are usually independent of attribute name semantics. It, therefore, makes sense to generalize such forms into the following Trace Schema forms, with ATIDs for Adjoint Threads of Activity for the first form, and with FIDs for Features of Activity for the second form:

Such Trace Schemas are useful for various trace and log joins other than Trace Mask.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When looking at trace and log messages we are usually interested in some features (for example, when doing feature engineering, but not limited to) which can be labelled via Feature IDs (FID). Messages that have the same FID value constitute Feature of Activity, similar to Thread of Activity (or Adjoint Thread of Activity).

Such Features of Activity can span several (A)TIDs in contrast to Fibers of Activity which are confined to the same (A)TID and may have different FID values. Therefore, inside (A)TID there can be several Features of Activity having different FID values.

This analysis pattern serves as a base for other data science analysis patterns we add next.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Intra-Correlation may be quite elaborate and include analysis of 2-dimensional Weaves of Activity. A similar 2-dimensional metaphor can be applied to Inter-Correlation between several artefacts such as traces and logs, configuration information including infrastructure as code (Small DA+TA), telemetry and event streams, memory dumps (Adjoint Spaces, Trace Presheaf, Memory Fibration, State Dump). All these memory patches, layers, and Trace Fabrics are “sewn” together by Braids, Threads, Adjoint Threads, Strands, Cords, and Weaves of Activities. We call this pattern Trace Quilt but analogy with quilting and quilts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Various metrics are covered by Counter Value trace and log analysis pattern. However, metric labels or metric metadata as implemented by monitoring tools such as Prometheus can be mapped directly to Adjoint Threads of Activity in our trace and log analysis pattern catalog:

<Metric Name>{<Label Name>=<Label Value>, ...}=<Metric Value> (from Prometheus data model)

{<Metric Name ATID>=<ATID Value>, <ATID Name>=<ATID Value>, ..., <Message (Metric Value)>}

This allows the application of many trace and log analysis patterns related to threading and adjoint threading (multibraiding).

We call this analysis pattern Message Metadata. It is illustrated for time series in the following diagram where we have the same Labels for all metric names (in general labels may be different):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The content of trace analysis patterns is available in edited and revised PDF format:

https://www.patterndiagnostics.com/trace-log-analysis-pattern-reference

Ideally a trace or log message should contain only one piece of information including associated data. However, some Multidimensional Messages may contain unrelated information, including several Message Invariants and variable data places, for example: “Entry GetData. Error opening file: 0×5″ or “Window handle: 0xa60834 pHandler: 0×456210F0″. Such messages may be split into several independent messages and, if necessary, additional ATIDs (new Adjoint Threads of Activity) may be added like depicted in this diagram of Combed Trace:

Another example is Exception Stack Trace messages in some logging implementations.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If messages from (Adjoint) Thread of Activity also have associated traces (Fiber Bundle) then the latter messages data, for example, module names, can be interlinked with corresponding Adjoint Threads of Activity, thus forming “two-dimensional” Weave of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace and log analysis patterns may be additionally applied not only to a database like tables but also to texts (as an example of general trace and log analysis). Sentences may form trace messages with paragraphs and chapters corresponding to traditional ATIDs (IDs for Adjoint Threads of Activity) such as TID and PID in the most simple syntax mapping case, and certain sentences may be interpreted as Silent Messages.

Different attribute generation schemas may be used, for example, selected vocabulary may be used to assign TID numbers. More complex cases may require paratexts, supplementary texts providing additional structure and semantic information like in the case of Paratext memory analysis pattern, the case of extended traces.

The opposite process of converting traces and logs to text is also possible with additional paratext generation if necessary. We call this two-way analysis pattern Text Trace. After converting texts to logs it is possible to apply the majority of trace and log analysis patterns from the catalog.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -