Software Diagnostics Library

A typical trace or log is a detailed narrative, and one of the first analysis tasks is to check whether the expected Basic Facts are present: user name, machine name, process, component, time, operation, resource, or other problem-description facts. If those facts are absent or inconsistent, the trace may have been collected at the wrong time, on the wrong system, or under different conditions:

However, vocabulary alone is not enough. A trace may contain all expected words and identifiers, yet the analyst (human or AI) may still not understand what kind of things they are, how they relate, and what diagnostic roles they play. Usernames, process IDs, request IDs, sessions, endpoints, file names, transactions, queues, locks, models, agents, function and tool calls, or error messages are trace entities with roles and relationships. Trace Ontology is a trace and log analysis pattern that identifies the domain entities, classes, relations, and constraints, both explicit and implicit, in trace messages. It turns raw trace vocabulary into a practical structured diagnostic model:

Therefore, Trace Ontology extends Basic Facts from vocabulary recognition to structured diagnostic representation. In summary, Basic Facts answer the following question: What vocabulary from the problem description is visible in the trace or log? Trace Ontology answers: What kinds of things exist in this trace, how are they related, and what can be inferred from those relations?

Also, Trace Ontology, as a representational model, can be coupled with Semantic Mapping, a presentation transformation that relabels opaque identifiers with meaningful names. Note that trace ontologies may exist before traces are collected. In such a case, they simplify the analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In trace and log analysis, inductance and capacitance can serve as metaphors for two distinct forms of diagnostic behavior.

Trace Inductance is the tendency of a system to resist sudden changes in behavior. A new input, configuration change, request burst, or failure condition may not immediately appear in the trace as a new stable pattern. The existing execution flow has “momentum.” Threads, queues, retries, caches, locks, connection pools, batching, and background workers continue to reflect the previous state for some time:

Note that the cause may not be visible in the trace or log but may come from another trace and log, similar to Paratext in memory analysis.

Trace Capacitance is the tendency of a system to accumulate diagnostic potential before a visible discharge occurs. The trace or log may look normal while internal state, queues, memory pressure, retry debt, latency, pending work, or error counters are accumulating. Then the system suddenly emits a burst of Error Messages, warnings, Timeouts, or Phase Transitions.

In summary, Trace Inductance explains delayed behavioral response, and Trace Capacitance explains delayed behavioral manifestation. The former asks: Why did the trace not change immediately after the cause? The latter asks: What was accumulating before the visible failure? Together they help avoid a common mistake: assuming that the first visible error is the real beginning of the problem. In many systems, the cause may appear before the symptom, because of inductance, and the symptom may appear suddenly because of capacitance.

We introduce Trace Reactance as a good umbrella analysis pattern name, with Trace Inductance and Trace Capacitance as two specializations of this pattern: Trace Reactance describes how diagnostic signals are delayed, smoothed, accumulated, or released by the system structure before becoming visible in traces and logs.

An agentic AI fits naturally here, too: agents accumulate context debt, token pressure, and retry state before a sudden degradation in output or a tool-call cascade both inductance and capacitance effects, for example (click on image to enlarge):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Large traces and logs often contain many combinations of conditions. The analyst sees many individual events but struggles to see which combinations are essential and which are redundant, equivalent, or adjacent manifestations of the same underlying behavior. The trace appears complex because the diagnostic space is fragmented into many small observations.

Karnaugh Map analysis pattern is useful when trace or log fragments can be classified by several binary or categorical dimensions, for example, distinctive features of Marked Messages. We project events into a structured logical space, similar to how a Karnaugh map projects Boolean combinations into an adjacency-preserving logical grid. Grouping cells in this grid simplifies the apparent complexity of many observed failure combinations into a minimal Boolean diagnostic condition, separating essential root cause dimensions from incidental ones that vary freely without affecting the outcome.

For example, suppose we analyze failures using four binary dimensions:

• A - Auth token expired
• B - Cache miss
• C - Backend timeout
• D - Retry attempt

We collected many traces for both working and non-working (failure) cases, and at first, it looks like there are four different failure cases:

But in Karnaugh-map form, these four cases form one group. The varying dimensions are B and D, while A and C remain constant:

So the simplified diagnostic condition is: failure occurs when the auth token expires and the backend times out, regardless of cache state or retry state. Or, in Boolean-like form: Failure = A ∧ C. This means cache misses and retries are not root discriminators here. They are incidental dimensions.

Here is a similar example for agentic AI:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Semantic Mapping is the trace and log analysis pattern where opaque runtime identifiers such as PIDs, TIDs, request IDs, handles, or session IDs are renamed or mapped to semantically meaningful diagnostic entities such as UI Thread, Worker Thread, Client Process, Blocking Thread, or Failed Request:

Additionally, in Semantic Mapping, we cannot only rename identifier values but also rename the Trace Schema itself, for example, changing column headers such as PID to Process and TID to Thread.

This analysis pattern differs from Trace Field, which is a mapping/function from trace messages to some other domain. It does not necessarily rewrite the trace presentation itself, but it may add additional ATID c to the Trace Schema. It is also different from Semantic Field, which is a semantic category/codomain/class into which trace messages are grouped, which is more about the meaningful domain of classification, not about rewriting trace labels. On the contrary, Semantic Mapping is a representation transformation that rewrites the trace into a more meaningful diagnostic form. It operates at two levels: instance level, renaming actual values, and schema level, renaming the fields/headers themselves.

Here is another example adapted to agentic AI:

Using mathematical analogies, Semantic Mapping is essentially a readability-preserving isomorphism: the structural information is unchanged, but a human (or AI analyst) now works in a named, meaningful coordinate system rather than an anonymous numeric one.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Usually, in traces and logs, messages from different components are highly interleaved. Direct chronological reading may cause confusion because every component’s Adjoint Thread of Activity appears entangled with every other one:

Bethe Ansatz analyzes such a trace by treating requests, threads, agents, transactions, or log-producing entities as “quasi-particles” whose global behavior can be reconstructed from many local pairwise interactions. The inspiration for the pattern name comes from Bethe ansatz, introduced by Hans Bethe in 1931. In physics, it is a method for constructing exact solutions of certain many-body systems: in integrable systems, complex many-body scattering can be represented through factorized two-body scattering processes. In trace analysis, we often face a “many-body” problem: many requests, threads, services, queues, locks, agents, retries, callbacks, and timeouts interact within a single shared diagnostic space. Instead of trying to understand the whole trace as one monolithic event cloud, we decompose it into stable activity lines and pairwise encounters that may explain the global behavior:

The Bethe ansatz has many forms, including coordinate, algebraic, analytic, functional, nested, and thermodynamic variants. For this pattern, the most useful metaphor is the coordinate Bethe ansatz: represent the global state by positions of entities and interaction effects between them. We have the following analogies:

• Particle/excitation: request, thread, transaction, agent, session, workflow
• Coordinate: timestamp, component, hop number, queue position, memory address, trace span
• Momentum/rapidity: (activity) rate, latency class, retry rhythm, priority, causal direction
• Two-body scattering: (pairwise interaction) lock contention, queue wait, API call, resource conflict
• Scattering phase shift: delay, reordered event, changed state, timeout extension, retry offset
• Factorized many-body scattering: whole trace explained as composition of pairwise effects
• Bethe equations: consistency constraints imposed by loops, boundaries, cycles, repeated paths
• Non-integrability: residual behavior not explainable by pairwise interactions

Interactions can be found among Motifs, Macrofunctions, and actors of Activity Theatre.

We suggest the following diagnostic analysis procedure:

1. Identify trace quasi-particles that preserve identity across the trace.
2. Choose a coordinate system: the trace can be read through coordinates other than time.
3. Detect pairwise encounters: look for places where two entities interact; these are diagnostic “scattering” events.
4. Estimate phase shifts where a pairwise encounter often changes the apparent trajectory of an activity. The phase shift is the observable deformation caused by interaction.
5. Test factorization by asking the question: Can the global anomaly be explained as a product of pairwise interactions?

If the answer to the last question is yes, then the system is “Bethe-like”: complex but decomposable. If the answer is no, there may be a true many-body effect, such as shared cache collapse, global scheduler starvation, cascading timeout storm, distributed deadlock, correlated retry amplification, emergent agentic loop, or resource exhaustion caused by collective behavior.

Additionally, we can form a Motivic Trace from the resulting pairwise interaction layers. Motivic Trace compresses a trace into explanatory motives. Bethe Ansatz compresses a trace into pairwise interaction motives whose composition reconstructs the observed global behavior. In this sense, Bethe Ansatz can be viewed as a structured route to Motivic Trace: first decompose the tangled chronological trace into stable activity lines and pairwise encounters; then integrate those encounters into higher-level explanatory motives such as queue delay, database lock wait, retry ordering, and response ordering. In summary: Motivic Trace is the broader compression pattern; Bethe Ansatz is a pairwise-factorized way to build it.

A historical note: this analysis pattern also extends physical analogies of debugging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Usually, software traces and logs are sorted by time.

Spatial Form is a specialized Sorted Trace in which trace or log messages are sorted by spatial, topological, or diagnostic proximity to a chosen origin component, device, process, service, or subsystem. Instead of reading the trace solely as a chronological sequence, we choose a diagnostic origin and arrange messages by their distance from that origin. Within each distance layer, the original local time order may still be preserved.

The pattern name comes from Joseph Frank’s The Idea of Spatial Form that is associated with reading narrative structure through juxtaposition and relational arrangement rather than only through linear chronological progression; the concept was introduced into literary discussion through his 1945 essay and later collected with reconsiderations in his book.

Sorted Trace is the more general pattern: messages are sorted according to some attribute value, for example, by TID, ATID, message type, message invariants, or message data.

For Spatial Trace, the distance may come from network topology, service dependency graph, component containment, process/thread ownership, device hierarchy, pipeline stage distance, proxy/gateway chain, address-space relation, storage or shard topology, causal adjacency, and many others. The resulting trace is not anti-temporal. It is spatially primary and temporally secondary.

Spatial Trace analysis pattern may help answer these questions: What happened around this proxy? Which nearby component first showed abnormal behavior? How did the request propagate outward? Was the fault local, adjacent, or remote? It may help distinguish local symptoms, adjacent symptoms, downstream effects, remote dependencies, and external causes. It gives the trace a layered diagnostic structure and spatial Layered Periodization. The pattern is therefore both a sorting technique and a reading strategy.

This pattern is especially useful for distributed systems, microservices, network devices, storage stacks, cloud control planes, request pipelines, proxies, gateways, and agentic AI workflows where activity is spread across many components.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Network is an analysis pattern in which traces and logs are treated as evidence for constructing an attributed interaction network N=(V, E), where vertices V are Motives, (Adjoint) Threads or Features of Activity, and their combinations, and directed edges E are created by an explicit correspondence rule between them, for example, request/response, causality, correlation propagation, spawn/join relation, or shared resource usage. A scope such as Time Delta or some filtering for Message Patterns may also be applied before the network construction.

Edge aggregation, weighting, and labels are part of the construction specification, so the result is not merely a drawing but a diagnostic network on which structural properties such as fan-in, fan-out, hubs, components, and derived measures such as Trace Divergence can be computed. This differs from Trace Graph, whose primary purpose is plotting or graphing trace data, and from Message Complex, whose primary elements are messages connected geometrically rather than identities connected relationally.

Trace Network analysis pattern differs from Causal History, Causal Messages, and Causal Chains in both primitive elements and construction intent. Causal History is a message-level structure whose arrows represent possible causation; Causal Messages are those messages selected as causally relevant within that history; and Causal Chains are abstractions of causal relations into linked 1-chains, 2-chains, and higher n-chains. By contrast, Trace Network is a general constructed network whose vertices are typically diagnostic identities rather than messages, and whose edges are induced by an explicitly declared relation derived from trace evidence, such as causal linkage, adjoint correspondence, request/response coupling, shared-resource mediation, or correlation transfer. Accordingly, a Trace Network may encode causal structure as one special case, but it is not restricted to causality and does not by itself imply chain-complex abstraction.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes, we want to count the number of (Adjoint) Threads of Activity corresponding to a specified ATID:

We can view this as (adjoint) threads coming into or out of the specified ATID, similar to divergence, which gives the name Trace Divergence log analysis pattern. This analysis pattern differs from Cord of Activity, which is not a number, and the latter may not have a single, unvarying source or target ATID to which other A(TID)s correspond. It is also different from Trace Flux, where the number of threads is an external variable not related to traces and logs, and from Message Flow, which operates on the individual message level, temporal in nature, and counters are set in advance.

Typical examples include SYN floods in network traces (src and dst ATIDs), the number of threads corresponding to the specific PID, or the number of threads contending for the specified API.

Activity Divergence may look similar, but its surface is temporal, whereas Trace Divergence’s, surface is structural. There can be several Trace Divergencies in the same trace or log since they are per ATID.

Formally, Trace Divergence is a property of a constructed graph, for example, D_in​(a)=∣{x∈V∣x→a}∣; Activity Divergence is a property of a constructed signal, interpreted as dynamics, for example, D_in​(a,t).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We write software based on requirements and then see its execution. The same analogy can be applied to Declarative Traces, which are “executed.” Trace Plans serve the role of tracing and logging requirements. The following diagram illustrates trace engineering and the lifecycle of tracing and logging:

We look at a resulting trace or log and relate it to its Trace Plan to find anomalies and problems not only in software execution but also in traces and logs themselves and improve tracing source code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we get traces and logs, we are interested in Trace Context: an issue description, how its trace was collected, overall system information, related Adjoint Spaces, Trace Summary, and previous traces and logs and their analyses. This contextual information can be organized as a checklist to ensure situational awareness, diagnostic quality, and reduce the number of information request roundtrips.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Message Embedding, as a representational technique in ML, are a variant of Trace Field. We can also consider the sequence of Message Embeddings as a trace itself with columns as latent features, forming separate latent Features of Activity. We can also treat these embeddings as sentence embeddings when interpreting traces and logs as Text Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can view traces and logs as abstract polynomials that consists of abstract monomials. For example, if we have trace messages A,B,C, and D, the trace ABCACACACCD represents a single monomial. The multiplication operation in monomials represents message concatenation. But we can also split the trace as an abstract sum of several monomials, for example, ABC + AC + AC + AC + CD, or ABC + 3*AC + CD. The addition operation is a concatenation of traces even if concatenated traces consist of just one message. Note the distinction here between concatenation of messages and traces. By Trace Polynomial we mean a canonical abstract polynomial representation where we divide the trace by monomial when the next message in the message stream is already contained in the previous monomial, for example, ABC + 2AC + AC^2D.

Both addition and multiplication are non-commutative, and no distributivity between them. Mathematically speaking, we have the so-called a non‑distributive bi‑semigroup, or, in a category-theoretic sense, such abstract polynomials are objects in a free 2‑semigroupal category without interchange.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Precision and Recall analysis patterns are borrowed from information retrieval and machine learning. In essence, Trace Precision is the fraction of trace messages used for successful diagnostics among all selected trace messages, for example, among CoTrace messages to reduce the noise. Trace Recall is the fraction of trace messages used for successful diagnostics among all relevant messages (these can also be messages not included in the trace, see Trace Bias).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Traces and logs can be considered models of computation as was first proposed in Trace Bias analysis pattern. Borrowing metaphors from machine learning we can consider trace messages as Trace Parameters having associated values, for example, through Trace Field. These values can be numerical weights or linguistic values, for example, as proposed by Zadeh. Message values can be field values themselves, for example, Counter Values. As was noted in Trace Variance, different tracing environments with their set of “features” can be considered input data. if we use an artificial neural network metaphor, trace analysis itself can be considered as an activation function. CoTraces, as models of trace analysis, may form other “hidden” layers with “backpropagation” updating message weights. Tracing configuration parameters including Trace Schemas can be considered as Trace Hyperparameters. These concepts can be roughly illustrated by the following diagram:

To reduce the number of Trace Parameters we can use Quotient Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In some cases, when we have two trace messages we can infer the missing Implicit Message. This trace augmentation may help when we Sparse Traces and we want to train ML models on them. To illustrate, we asked ChatGPT:

We have a log file with two messages: 1. Reading file test.txt 2. Access denied. There is an implicit message in between.

The implicit message between the two log entries is:

“Attempting to open file ‘test.txt’.”

Reasoning:

1. “Reading file test.txt” → This suggests that the system or user has issued a command to read the file.
2. Implicit step (not logged explicitly) → The system attempts to open the file to read its contents.
3. “Access denied.” → The attempt fails due to permission restrictions or security policies.

Thus, the missing step is the system trying to open or access the file, which leads to the access denial.

Implicit Messages are different from Silent Messages which are padding messages to occupy the minimum resolution gaps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The fact that messages have types is mentioned in some trace and log analysis patterns but have never had its own analysis pattern. Trace and log messages may have the same Message Type if they have the same Adjoint Thread of Activity ATID (or Message Metadata) value, for example, “Warning”. Or they may have the same structure, for example, the same constants or message invariants, or belong to the same Trace Schema in Container Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Sketch can have several Trace Models (borrowed from model theory with sketches representing the logic of traces and logs) when messages satisfy trace and log analysis patterns sketched in Dia|gram language illustrations:

The same Trace Sketch induces an equivalence relation between different Trace Models, yet another Trace Similarity measure. Also, models of traces and logs having the same Trace Shapes may not be equivalent.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Sketch embodies Dia|gram language approach: in essence each trace and log analysis pattern illustration is a sketch. For example, a WinDbg log is represented as sequence of different Activity Regions:

Another example of Trace Sketch is Trace Skeleton.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Feynman Trace borrows ideas from the path integral formulation of quantum mechanics. Such a trace includes all possible traces from all possible traces generated from all possible executions including Trace Amplitudes, Empty Traces, Use Case Trails, and traces with Error Messages, but excluding Impossible Traces:

Code flow Declarative Trace analysis can be used to assess the relative contributions of trace and log variants. To reduce infinities arising from loops, Renormalization can be used.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When comparing different traces from the same system we may see different correlations of Statement Densities. For example, when A message density is increased, then C message density is also increased regardless of any changes to B message density. We can borrow concentration notation from chemical kinetics and use [A], [B], and [C] for corresponding Statement Densities, either local in Activity Regions or globally for the whole trace. Observed correlations may point to existing causal mechanisms (like when kinetics points to reaction mechanisms):

This Message Kinetics pattern is more general than Relative Density where a semantic relationship is already known and the comparison is made between working and non-working scenarios. Variations of message densities may occur in normal scenarios, for example, with different amount of input data. There can be several types of Message Kinetics in one trace or log.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -