Software Diagnostics Library

Even when all relevant events are present in traces and logs, they may still remain opaque because the analyst lacks the specialized skill needed to interpret their messages, abstractions, timing, domain vocabulary, failure semantics, or their role in the broader Trace World:

Trace Skill is a trace and log analysis pattern in which successful interpretation depends on a specific skill set required by the trace domain or Trace Ontology:

Some traces can be read with general diagnostic experience, especially if domain-independent analysis patterns are used. Others require specialist knowledge:

• Kernel tracing requires OS internals skill.
• Distributed traces require distributed systems skill.
• Database logs require query, transaction, and storage skill.
• Security logs require identity, policy, token, and threat-modeling skill.
• ML/AI traces require model, prompt, embedding, evaluation, and agentic workflow skill.
• Performance traces require latency, contention, scheduling, and resource-analysis skill.
• Business-process traces require domain workflow knowledge.

The same trace may therefore be simple for one analyst and almost invisible for another. Even the same analysis pattern may be applied differently. The problem is not the absence of data but the absence of the interpretive skill needed to turn trace data into diagnostic meaning:

Trace Skill analysis pattern may be useful in the following situations:

• The trace contains relevant evidence, but the analyst does not recognize it.
• Messages are dismissed as noise because their domain meaning is unknown.
• A team reads only the parts of the trace matching its own expertise.
• Different specialists disagree because each understands only one layer.
• The trace crosses technical boundaries: application, infrastructure, security, data, ML, business logic.
• The incident is repeatedly escalated because no one possesses the required combination of skills.

Compared to other analysis patterns, Trace Viewpoints are perspectives from which the trace is read. Trace Skill is the competence needed to read from a viewpoint correctly. So a viewpoint can be chosen quickly, but a skill has to be possessed, acquired, or borrowed from someone else. Trace World allows movement across skill sets by creating a common diagnostic language, but Trace Skill explains why this is necessary: traces are often multi-skill artifacts. No single narrow skill may be enough. The analyst either needs multiple skills or needs to collaborate across skill boundaries using a shared Trace World.

We can also say that Trace Skill is needed to navigate traces and logs and their Trace World. Trace World provides the shared diagnostic terrain constructed from trace and log evidence, entities, relations, events, timelines, and viewpoints. Trace Skill is the specialized competence required to navigate both the raw evidence and the reconstructed world accurately, recognize relevant signals, switch between viewpoints, and avoid misinterpretation.

In summary, Trace Skill is a trace and log analysis pattern in which decisive diagnostic evidence can only be recognized by the appropriate specialist skill set.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A single trace may reveal only a partial diagnostic world. An application trace may show requests and exceptions. A security trace may show tokens, scopes, and access decisions. An operations trace may show retries, timeouts, latency, and resource pressure. An AI/ML trace may show prompts, tool calls, agent decisions, model responses, and memory updates:

However, different traces are not only different in content. They may also belong to different Implementation Discourses. The earlier Implementation Discourse analysis pattern observes that non-trivial traces contain different discourses because components are written in different languages and follow different runtime environments, binary models, and interface frameworks. These implementation variations influence the structure, syntax, and semantics of trace messages; for example, .NET traces differ from file system driver traces or COM debugging messages:

Trace World builds on this idea: it is a trace and log analysis pattern in which multiple traces, logs, telemetry streams, and diagnostic Trace Viewpoints contribute to a shared ontology of entities, relations, events, and narratives. The key idea is that multiple traces do not merely sit side by side. They share and enrich the same diagnostic ontology. As more traces are added, the Trace World becomes richer: new entities appear, existing entities are connected, relations become clearer, and the common narrative becomes more complete:

Therefore, Implementation Discourse is the local linguistic form of trace evidence, while Trace World is the shared world model that allows those local discourses to be translated, aligned, and used together. Implementation Discourse: How this component, runtime, framework, or language speaks in the trace. Trace World: How many such trace languages contribute to one shared diagnostic world. This also clarifies the relationship to Trace Ontology, which extracts entities, events, and relations from a trace, whereas Trace World allows multiple traces and multiple implementation discourses to share and enrich the same ontology.

Compared to Trace Viewpoints, which are different ways of reading the same trace world, and Implementation Discourses, which are different trace languages used to express evidence, Trace World is the common diagnostic world that persists when we move across both viewpoints and implementation discourses:

In agentic AI systems, this becomes especially important. Prompt traces, tool-call traces, memory traces, policy traces, model inference traces, application logs, and infrastructure telemetry may all speak different implementation discourses. Trace World provides the common diagnostic world where agents, tools, prompts, observations, memories, users, services, policies, and failures can be connected.

In summary, Trace World is the shared diagnostic world formed when multiple trace discourses enrich the same ontology, allowing analysts to move across viewpoints without losing entity identity, relations, or narrative continuity.

This analysis pattern is useful when:

• Several traces describe the same incident from different systems.
• Components use different trace languages, formats, and conventions.
• Different runtimes or frameworks produce structurally different messages, including Embedded Traces.
• The same entity appears under different names in different traces.
• A failure crosses application, infrastructure, security, data, or AI boundaries.
• A single trace discourse is insufficient to reconstruct the diagnostic story.

For constructing the Trace World, typical analysis steps may be these:

1. Identify the Implementation Discourse of each trace.
2. Extract Basic Facts from each discourse.
3. Identify local entities, events, and relations.
4. Map equivalent entities across discourses.
5. Merge them into a shared Trace World ontology.
6. Add relations that become visible only across traces.
7. Build a common narrative across implementation boundaries.
8. Use Trace Viewpoints to read the same world from different skill perspectives.

However, there might be some problems when using Trace World analysis pattern:

• Discourse isolation: each trace language remains separate.
• Translation gap: no mapping exists between local trace terms.
• Identity mismatch: the same entity has different names in different discourses.
• Semantic drift or incompatible Semantic Mappings: similar words mean different things in different traces.
• Partial or incompatible Trace Ontologies: one discourse lacks entities needed to explain another.
• Conflicting narrative: traces imply different causal stories.
• Unenriched world: traces are collected but not integrated.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A typical trace or log is a detailed narrative, and one of the first analysis tasks is to check whether the expected Basic Facts are present: user name, machine name, process, component, time, operation, resource, or other problem-description facts. If those facts are absent or inconsistent, the trace may have been collected at the wrong time, on the wrong system, or under different conditions:

However, vocabulary alone is not enough. A trace may contain all expected words and identifiers, yet the analyst (human or AI) may still not understand what kind of things they are, how they relate, and what diagnostic roles they play. Usernames, process IDs, request IDs, sessions, endpoints, file names, transactions, queues, locks, models, agents, function and tool calls, or error messages are trace entities with roles and relationships. Trace Ontology is a trace and log analysis pattern that identifies the domain entities, classes, relations, and constraints, both explicit and implicit, in trace messages. It turns raw trace vocabulary into a practical structured diagnostic model:

Therefore, Trace Ontology extends Basic Facts from vocabulary recognition to structured diagnostic representation. In summary, Basic Facts answer the following question: What vocabulary from the problem description is visible in the trace or log? Trace Ontology answers: What kinds of things exist in this trace, how are they related, and what can be inferred from those relations?

Also, Trace Ontology, as a representational model, can be coupled with Semantic Mapping, a presentation transformation that relabels opaque identifiers with meaningful names. Note that trace ontologies may exist before traces are collected. In such a case, they simplify the analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In trace and log analysis, inductance and capacitance can serve as metaphors for two distinct forms of diagnostic behavior.

Trace Inductance is the tendency of a system to resist sudden changes in behavior. A new input, configuration change, request burst, or failure condition may not immediately appear in the trace as a new stable pattern. The existing execution flow has “momentum.” Threads, queues, retries, caches, locks, connection pools, batching, and background workers continue to reflect the previous state for some time:

Note that the cause may not be visible in the trace or log but may come from another trace and log, similar to Paratext in memory analysis.

Trace Capacitance is the tendency of a system to accumulate diagnostic potential before a visible discharge occurs. The trace or log may look normal while internal state, queues, memory pressure, retry debt, latency, pending work, or error counters are accumulating. Then the system suddenly emits a burst of Error Messages, warnings, Timeouts, or Phase Transitions.

In summary, Trace Inductance explains delayed behavioral response, and Trace Capacitance explains delayed behavioral manifestation. The former asks: Why did the trace not change immediately after the cause? The latter asks: What was accumulating before the visible failure? Together they help avoid a common mistake: assuming that the first visible error is the real beginning of the problem. In many systems, the cause may appear before the symptom, because of inductance, and the symptom may appear suddenly because of capacitance.

We introduce Trace Reactance as a good umbrella analysis pattern name, with Trace Inductance and Trace Capacitance as two specializations of this pattern: Trace Reactance describes how diagnostic signals are delayed, smoothed, accumulated, or released by the system structure before becoming visible in traces and logs.

An agentic AI fits naturally here, too: agents accumulate context debt, token pressure, and retry state before a sudden degradation in output or a tool-call cascade both inductance and capacitance effects, for example (click on image to enlarge):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -