Software Diagnostics Library

(Adjoint) Threads of Activity can be interpreted as braids (multibraiding). This braid analogy assumes that all (adjoint) threads implicitly start and end outside of the trace boundaries. However, some (adjoint) threads may start after the beginning of the tracing or end before the finishing of the tracing. Such modified braids are called braidoids. There can be several braidoids per trace based on the chosen (A)TIDs. We call this analysis pattern Trace Braidoids and one, based on TID, is illustrated in the following diagram:

We added arc crossings when a different TID becomes current. Please also compare these crossings with other analysis patterns such as Braid Group and Braid of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Message data may point to other messages in the same trace (see the example of Linked Messages) or in the other trace (see the Data Selector example). But similar data in other messages may not point to any other messages in the same or other, perhaps Truncated, traces and logs collected at the same time - similar to invalid pointers, for example, kernel addresses in process memory dumps or user space addresses in kernel memory dumps. We call this analysis pattern Null Reference (also notice the analogy with foreign key values in data tables where Null is not a value). Another analogy here is referential failure. This is illustrated in the following diagram adapted from Linked Messages analysis pattern diagram.

This analysis pattern is different from Missing Data where the reference is itself missing.

These Null References can be remediated by longer supplemental traces, Fiber Bundle, and Adjoint Spaces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we disregard the length of message blocks having the same attribute (ATID), we get Quotient Trace. But when we disregard the content of these message blocks (and replace each message with the same “empty” non-Silent Message) but preserve their length, we get Trace Skeleton.

Different Trace Shapes may have different Trace Skeletons but we can generate similar shapes from one skeleton.

We can also apply a music metaphor and consider it as Trace Rhythm:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we want to collapse messages into one message while preserving content, for example, for grep. We call such an analysis pattern Collapsed Message (by an analogy for collapsing an internal directed graph edge, or edge contraction). Several consecutive messages having the same attribute, for example Thread of Activity for Exception Stack Trace may be collapsed into one longer trace message. This is a simple case of Quotient Trace without compression or transform. And this is different than Motivic Trace which doesn’t preserve message content.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Various types of Measurements are important in software diagnostics. We consider traces and logs, and general software narratives (and even hardware narratives) as a medium for all types of possible measurements. Even a small display in a handheld device showing a number is an example of Singleton Trace.

Typical trace and log measurement analysis patterns include Time Delta, Statement Density and Current, and Trace Acceleration. Numeric analysis patterns include Counter Value, Trace Field, Signal in general, and the forthcoming Trace Distance that uses various metrics, for example, the number of messages, Activity Regions, or just hops.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Even if traces and logs are perfectly synchronized (Unsynchronized Traces) we me still get Unsynchronized Messages. If respective message times are the same (can depend on time resolution) we don’t know which one was first. This is visible if we do Trace Mask using different order:

If both messages belong to the same Thread of Activity we may be able to reorder them correctly based on additional message semantics, such as module hierarchy (for example, OS runtime library and application code that are traced separately).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Usually, when we find an interesting message in a log (maybe also a frame from Exception Stack Trace), especially from an unfamiliar component, we also want to search past problem cases either on the Internet or in some internal database. However, we just put the message as is or some small fragment of it we may get a lot of noise results. The problem is to find the optimal Message Essence. Often, this is done by omitting variable data (including Adjoint Thread of Activity fields) but leaving Message Invariants and Trace Constants usually refine a diagnostic error:

This analysis pattern is different from Message Invariant. where the latter is useful when finding its emitter’s source code lines (PLOT).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we have different traces and logs not necessarily with the same Trace Schema and select only messages that have some condition, for example, the same ATID (see Adjoint Thread of Activity) or FID (see Feature of Activity) value, we get the new trace that we call Trace Join. A combination of ATID from one trace or Message Set from another is also possible as illustrated in this allegorical picture when joining is done by “Plato” author value or title containing “Plato” (all case-insensitive):

This is very similar to relational data joins. Join of the same trace is possible too. A Dia|gram picture (similar to the previous patterns) is left as an exercise.

We initially wanted to call this analysis pattern Filtered Mask but later realized that it may not be possible to do Trace Mask if there is no global ordering information, such as time. In such a case, Serial Trace is possible. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The name of this pattern, Container Trace, originates from the logging style that is recommended as the best practice for containers (like Docker), where various components output their tracing and logging statements to the standard console output. Such components may have their own incompatible Trace Schemas, for example, normal trace messages intermingled with Exception Stack Traces. However, in general, this pattern can be extended to any log file (a container for trace statements). This pattern is different from Trace Mask, where individual traces come from separate files and have Trace Schema with some ATID (see Adjoint Thread of Activity) or FID (see Feature of Activity), such as time, that allows for blending them correctly. Components that output their messages to Container Trace may not even have any internal Trace Schema. In such a case, Container Trace may simply be treated as Text Trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Most of the time tracing and logging is done sequentially, for example, when a service or application is restarted after the crash or bug fix, or the host is rebooted. Then we can glue all resulted traces together (similar to Glued Stack Trace) into one large Serial Trace. Here we assume the same Trace Schema for all individual traces and logs. It can also be considered as flattening a 2-dimensional Trace Tensor:

This allows us to apply various trace and log analysis patterns to the unified Serial Trace instead of doing Inter-Correlation (vs. Intra-Correlation).

Serial Trace is different from Meta Trace which is a trace about trace and Master Trace which is a trace we compare all other traces to. It is similar to Trace Mask when there is no overlap in time. Also, Serial Trace is not a reverse of Split Trace in a general case due to Visibility Limits between individual traces.

When gluing traces together, Ornament messages may be added to serve as a boundary between fragments.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Schema can be represented as Schema Trace or, avoiding naming confusion, Definition Trace. The resulting trace looses ordering (similar to unordered Message Set) but allows application of trace and log analysis patterns, especially if some order is fixed, for example, alphabetical for names or original presentation column arrangement. Schema definition Trace Schema can be represented as another Definition Trace as illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Most of trace and log analysis pattern illustrations using Dia|gram language are of these two general forms:

Although the first form represents typical ETW trace attributes, the analysis pattern descriptions are usually independent of attribute name semantics. It, therefore, makes sense to generalize such forms into the following Trace Schema forms, with ATIDs for Adjoint Threads of Activity for the first form, and with FIDs for Features of Activity for the second form:

Such Trace Schemas are useful for various trace and log joins other than Trace Mask.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When looking at trace and log messages we are usually interested in some features (for example, when doing feature engineering, but not limited to) which can be labelled via Feature IDs (FID). Messages that have the same FID value constitute Feature of Activity, similar to Thread of Activity (or Adjoint Thread of Activity).

Such Features of Activity can span several (A)TIDs in contrast to Fibers of Activity which are confined to the same (A)TID and may have different FID values. Therefore, inside (A)TID there can be several Features of Activity having different FID values.

This analysis pattern serves as a base for other data science analysis patterns we add next.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Intra-Correlation may be quite elaborate and include analysis of 2-dimensional Weaves of Activity. A similar 2-dimensional metaphor can be applied to Inter-Correlation between several artefacts such as traces and logs, configuration information including infrastructure as code (Small DA+TA), telemetry and event streams, memory dumps (Adjoint Spaces, Trace Presheaf, Memory Fibration, State Dump). All these memory patches, layers, and Trace Fabrics are “sewn” together by Braids, Threads, Adjoint Threads, Strands, Cords, and Weaves of Activities. We call this pattern Trace Quilt but analogy with quilting and quilts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Various metrics are covered by Counter Value trace and log analysis pattern. However, metric labels or metric metadata as implemented by monitoring tools such as Prometheus can be mapped directly to Adjoint Threads of Activity in our trace and log analysis pattern catalog:

<Metric Name>{<Label Name>=<Label Value>, ...}=<Metric Value> (from Prometheus data model)

{<Metric Name ATID>=<ATID Value>, <ATID Name>=<ATID Value>, ..., <Message (Metric Value)>}

This allows the application of many trace and log analysis patterns related to threading and adjoint threading (multibraiding).

We call this analysis pattern Message Metadata. It is illustrated for time series in the following diagram where we have the same Labels for all metric names (in general labels may be different):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The content of trace analysis patterns is available in edited and revised PDF format:

https://www.patterndiagnostics.com/trace-log-analysis-pattern-reference

Abstract is usually the summary of an artifact (see Trace Summary analysis pattern) or not concrete description (see Analysis Pattern Square diagram).

Accent as stress in a line of verse has its correspondence to data in Message Pattern, which can be seen as a sequence of variables and Message Invariants.

Act as a play division corresponds to Activity Regions (see also trace partitioning and Activity Theatre analysis patterns).

Action as the main story of a narrative artifact may involve a sequence of selected Significant Events, Macrofunctions, Activity Regions with Motives. In a software narratological framework for presenting software stories, action is a sequence of selected messages that constitutes a software plot (an acquired software artifact that may not be complete/full due to abridgment like restricting tracing/logging to selected components).

Adaptation as interpreting an artifact as a different one (from one media to another, or a different structure) is similar to treating memory dumps as traces/logs or vice versa as Projective Debugging.

Address as a story written for a specific group of people could be a software execution artifact explicitly acquired and adapted to some external users or Declarative Trace messages crafted for a specific team in mind (see also Embedded Comment analysis pattern).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Ab ovo is a software story (for example, a trace or log, a problem description, see software narratology square) that starts from the beginning of the use case events it narrates (see also Use Case Trail analysis patterns) or the start of software execution (see also Visibility Limit analysis pattern). Logging may start from some middle event of a use case, source code (see also Declarative Trace analysis pattern), or a log may be a part of a larger full trace (see also a software narratological framework for presenting software stories): in medias res. Such software stories may also have flashbacks, for example, stack traces, especially in software problem descriptions. Often, flashbacks are the only available software stories. Some tracing and logging sessions may be deliberately shortened to save space, communication throughput, or other reasons like security, similar to abridged editions of literary works (see also Abridged Dump and Missing Component analysis patterns). Such editions of software execution artifacts often hinder analysis (see Lateral Damage analysis pattern).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Ideally a trace or log message should contain only one piece of information including associated data. However, some Multidimensional Messages may contain unrelated information, including several Message Invariants and variable data places, for example: “Entry GetData. Error opening file: 0×5″ or “Window handle: 0xa60834 pHandler: 0×456210F0″. Such messages may be split into several independent messages and, if necessary, additional ATIDs (new Adjoint Threads of Activity) may be added like depicted in this diagram of Combed Trace:

Another example is Exception Stack Trace messages in some logging implementations.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If messages from (Adjoint) Thread of Activity also have associated traces (Fiber Bundle) then the latter messages data, for example, module names, can be interlinked with corresponding Adjoint Threads of Activity, thus forming “two-dimensional” Weave of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -