Software Diagnostics Library

Trace Network is an analysis pattern in which traces and logs are treated as evidence for constructing an attributed interaction network N=(V, E), where vertices V are Motives, (Adjoint) Threads or Features of Activity, and their combinations, and directed edges E are created by an explicit correspondence rule between them, for example, request/response, causality, correlation propagation, spawn/join relation, or shared resource usage. A scope such as Time Delta or some filtering for Message Patterns may also be applied before the network construction.

Edge aggregation, weighting, and labels are part of the construction specification, so the result is not merely a drawing but a diagnostic network on which structural properties such as fan-in, fan-out, hubs, components, and derived measures such as Trace Divergence can be computed. This differs from Trace Graph, whose primary purpose is plotting or graphing trace data, and from Message Complex, whose primary elements are messages connected geometrically rather than identities connected relationally.

Trace Network analysis pattern differs from Causal History, Causal Messages, and Causal Chains in both primitive elements and construction intent. Causal History is a message-level structure whose arrows represent possible causation; Causal Messages are those messages selected as causally relevant within that history; and Causal Chains are abstractions of causal relations into linked 1-chains, 2-chains, and higher n-chains. By contrast, Trace Network is a general constructed network whose vertices are typically diagnostic identities rather than messages, and whose edges are induced by an explicitly declared relation derived from trace evidence, such as causal linkage, adjoint correspondence, request/response coupling, shared-resource mediation, or correlation transfer. Accordingly, a Trace Network may encode causal structure as one special case, but it is not restricted to causality and does not by itself imply chain-complex abstraction.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes, we want to count the number of (Adjoint) Threads of Activity corresponding to a specified ATID:

We can view this as (adjoint) threads coming into or out of the specified ATID, similar to divergence, which gives the name Trace Divergence log analysis pattern. This analysis pattern differs from Cord of Activity, which is not a number, and the latter may not have a single, unvarying source or target ATID to which other A(TID)s correspond. It is also different from Trace Flux, where the number of threads is an external variable not related to traces and logs, and from Message Flow, which operates on the individual message level, temporal in nature, and counters are set in advance.

Typical examples include SYN floods in network traces (src and dst ATIDs), the number of threads corresponding to the specific PID, or the number of threads contending for the specified API.

Activity Divergence may look similar, but its surface is temporal, whereas Trace Divergence’s, surface is structural. There can be several Trace Divergencies in the same trace or log since they are per ATID.

Formally, Trace Divergence is a property of a constructed graph, for example, D_in​(a)=∣{x∈V∣x→a}∣; Activity Divergence is a property of a constructed signal, interpreted as dynamics, for example, D_in​(a,t).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We write software based on requirements and then see its execution. The same analogy can be applied to Declarative Traces, which are “executed.” Trace Plans serve the role of tracing and logging requirements. The following diagram illustrates trace engineering and the lifecycle of tracing and logging:

We look at a resulting trace or log and relate it to its Trace Plan to find anomalies and problems not only in software execution but also in traces and logs themselves and improve tracing source code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we get traces and logs, we are interested in Trace Context: an issue description, how its trace was collected, overall system information, related Adjoint Spaces, Trace Summary, and previous traces and logs and their analyses. This contextual information can be organized as a checklist to ensure situational awareness, diagnostic quality, and reduce the number of information request roundtrips.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Message Embedding, as a representational technique in ML, are a variant of Trace Field. We can also consider the sequence of Message Embeddings as a trace itself with columns as latent features, forming separate latent Features of Activity. We can also treat these embeddings as sentence embeddings when interpreting traces and logs as Text Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can view traces and logs as abstract polynomials that consists of abstract monomials. For example, if we have trace messages A,B,C, and D, the trace ABCACACACCD represents a single monomial. The multiplication operation in monomials represents message concatenation. But we can also split the trace as an abstract sum of several monomials, for example, ABC + AC + AC + AC + CD, or ABC + 3*AC + CD. The addition operation is a concatenation of traces even if concatenated traces consist of just one message. Note the distinction here between concatenation of messages and traces. By Trace Polynomial we mean a canonical abstract polynomial representation where we divide the trace by monomial when the next message in the message stream is already contained in the previous monomial, for example, ABC + 2AC + AC^2D.

Both addition and multiplication are non-commutative, and no distributivity between them. Mathematically speaking, we have the so-called a non‑distributive bi‑semigroup, or, in a category-theoretic sense, such abstract polynomials are objects in a free 2‑semigroupal category without interchange.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Precision and Recall analysis patterns are borrowed from information retrieval and machine learning. In essence, Trace Precision is the fraction of trace messages used for successful diagnostics among all selected trace messages, for example, among CoTrace messages to reduce the noise. Trace Recall is the fraction of trace messages used for successful diagnostics among all relevant messages (these can also be messages not included in the trace, see Trace Bias).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Traces and logs can be considered models of computation as was first proposed in Trace Bias analysis pattern. Borrowing metaphors from machine learning we can consider trace messages as Trace Parameters having associated values, for example, through Trace Field. These values can be numerical weights or linguistic values, for example, as proposed by Zadeh. Message values can be field values themselves, for example, Counter Values. As was noted in Trace Variance, different tracing environments with their set of “features” can be considered input data. if we use an artificial neural network metaphor, trace analysis itself can be considered as an activation function. CoTraces, as models of trace analysis, may form other “hidden” layers with “backpropagation” updating message weights. Tracing configuration parameters including Trace Schemas can be considered as Trace Hyperparameters. These concepts can be roughly illustrated by the following diagram:

To reduce the number of Trace Parameters we can use Quotient Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In some cases, when we have two trace messages we can infer the missing Implicit Message. This trace augmentation may help when we Sparse Traces and we want to train ML models on them. To illustrate, we asked ChatGPT:

We have a log file with two messages: 1. Reading file test.txt 2. Access denied. There is an implicit message in between.

The implicit message between the two log entries is:

“Attempting to open file ‘test.txt’.”

Reasoning:

1. “Reading file test.txt” → This suggests that the system or user has issued a command to read the file.
2. Implicit step (not logged explicitly) → The system attempts to open the file to read its contents.
3. “Access denied.” → The attempt fails due to permission restrictions or security policies.

Thus, the missing step is the system trying to open or access the file, which leads to the access denial.

Implicit Messages are different from Silent Messages which are padding messages to occupy the minimum resolution gaps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The fact that messages have types is mentioned in some trace and log analysis patterns but have never had its own analysis pattern. Trace and log messages may have the same Message Type if they have the same Adjoint Thread of Activity ATID (or Message Metadata) value, for example, “Warning”. Or they may have the same structure, for example, the same constants or message invariants, or belong to the same Trace Schema in Container Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Sketch can have several Trace Models (borrowed from model theory with sketches representing the logic of traces and logs) when messages satisfy trace and log analysis patterns sketched in Dia|gram language illustrations:

The same Trace Sketch induces an equivalence relation between different Trace Models, yet another Trace Similarity measure. Also, models of traces and logs having the same Trace Shapes may not be equivalent.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Sketch embodies Dia|gram language approach: in essence each trace and log analysis pattern illustration is a sketch. For example, a WinDbg log is represented as sequence of different Activity Regions:

Another example of Trace Sketch is Trace Skeleton.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Feynman Trace borrows ideas from the path integral formulation of quantum mechanics. Such a trace includes all possible traces from all possible traces generated from all possible executions including Trace Amplitudes, Empty Traces, Use Case Trails, and traces with Error Messages, but excluding Impossible Traces:

Code flow Declarative Trace analysis can be used to assess the relative contributions of trace and log variants. To reduce infinities arising from loops, Renormalization can be used.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When comparing different traces from the same system we may see different correlations of Statement Densities. For example, when A message density is increased, then C message density is also increased regardless of any changes to B message density. We can borrow concentration notation from chemical kinetics and use [A], [B], and [C] for corresponding Statement Densities, either local in Activity Regions or globally for the whole trace. Observed correlations may point to existing causal mechanisms (like when kinetics points to reaction mechanisms):

This Message Kinetics pattern is more general than Relative Density where a semantic relationship is already known and the comparison is made between working and non-working scenarios. Variations of message densities may occur in normal scenarios, for example, with different amount of input data. There can be several types of Message Kinetics in one trace or log.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Variance is the measure of variability of traces and logs in different environments. This trace and log analysis pattern metaphor is borrowed from variance in machine learning like Trace Bias. Here, different environments or the same environment at different times are considered as input data.

Other analysis patterns can be used to investigate Trace Variance such as Bifurcation Point, Trace String, and Polytrace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Bias is the measure of how faithfully traces and logs as a models of computation reflect the actual computation. The term bias has many meanings. This trace and log analysis pattern metaphor is borrowed from model bias in machine learning and statistics. Traces, and therefore, their biases range from Empty Traces to Sparse Traces to traces from Time Travel Debugging tools. How we do tracing, for example, via Declarative Traces or Moduli Traces, have direct impact on Trace Bias.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Lattice is a selection of messages based on a fixed order distance between them (similar to lattices in geometry, one-dimensional in this case) or some other metric:

This analysis pattern is different from Time Scale where fixed time distance is used with additional analysis transformations.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Pressure is a “thermodynamic” variable along with Trace Temperature and Trace Volume. The fourth variable, the number of non-Silent Messages, N, is obvious. In the following diagram, two parts of the trace have approximately the same volume but different “temperature” that result in approx. same “pressure” according to a metaphor of an ideal gas law:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Temperature is an external quantitative (a number) or qualitative (for example, “hot”, “warm”, or “cold”) trace attribute of a trace or log or its fragment such as Activity Region that measures importance of the incident:

It is one example of Trace Field. Being an external attribute, independent from content, the trace with very few or no messages (Sparse Trace) can still have very high “temperature.” However, this analysis pattern is not the same as News Value which is an internal attribute based on trace content that measures its importance.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Volume counts not only messages but also Silent Messages:

The volume metaphor can also be applied to parts of the trace such as Activity Regions, Threads (and Adjoint Threads) of Activity, and various braids.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -