Software Diagnostics Library

Trace Similarity analysis pattern uses various similarity measures to assess the closeness of one trace or log to another. Here we provide an illustrative example using Jaccard index. Consider three simple logs where sample sets consist from Activity Regions:

The following table shows calculation of similarity between A and B, A and C, and B and C:

It’s possible to use sample sets consisting of messages instead. For our toy example we get similar index numbers:

We get different indexes though for individual regions and messages, for example:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to Declarative Trace we have code statements that may be intercepted by external API tracing tools (CreateFile, CloseHandle):

In the sample code above we have different logs resulted from Declarative Trace (DebugLog, OutputDebugString):

In addition, we have a log saved by an external tracing tool (for example, Process Monitor) that includes our API calls:

All such trace-generating source code statements form Moduli Trace as soon as they are executed:

Such a trace can also be analyzed using trace and log analysis patterns like other trace types. We took the idea of this analysis pattern from moduli spaces in mathematics that parametrize other spaces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Cartesian Trace analysis pattern has its analogical roots in Cartesian product. It covers a case where we have a long trace and a few Small DA+TA configuration traces (files). The former trace messages are associated with the latter messages (content or content changes) as depicted in the following diagram:

Think about a rectangle as a product of two line fragments or a cylinder as a product of a circle and a line fragment. Both traces are completely independent in comparison to Fiber Bundle, Trace Presheaf, or Trace Extension.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Some trace and log messages may have different grammatical structure and content but similar semantics. Therefore, we can create a table listing equivalent messages (using some equivalence relation) and use it to construct simpler traces and logs as depicted in this picture:

One trivial example of Equivalent Messages analysis pattern is Quotient Trace. Another example is Inter-Correlational analysis of logs that have different structure and format. In such a case Equivalent Messages simplify the analysis of higher Trace Dimensions.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If we run software with its default configuration and no interaction (input data) we get its Minimal Trace:

Such traces may have their own Master Trace. Also, Minimal Trace is a specific Use Case Trail. Metaphorically, they can be considered as minimal surfaces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Based on a mathematical analogy with critical points in topology (Morse theory) we introduce Critical Points in trace and log analysis where they signify the change of trace or log “shape” (topological or “geometric” properties) as illustrated in the following diagram:

Such a point may be an individual message, its Message Context, or Activity Region.

Critical Points are examples of Intra-Correlation whereas Bifurcation Points are examples of Inter-Correlation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes, we ask for a log file to see State and Event pattern, and see it there, only to find that we cannot do Back Trace of State Dumps from some Significant Event for Inter-Correlation analysis because our Data Interval is truncated (Truncated Trace). This highlights the importance of proper tracing intervals that we call Significant Interval analysis pattern by analogy with significant digits in scientific measurements. The following diagram illustrate the pattern:

If you find out you get truncated traces and logs often you may want to increase Statement Current for state logging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can “integrate” trace message stream into another, smaller trace. By analogy with motivic integration in contemporary mathematics we call this analysis pattern Motivic Trace. There can be border cases where the whole trace is reduced to one message or every message is associated with a different message (perhaps shorter or a number). Message Sets that are integrated into Motivic Trace can be completely different (for example, based on Motives) in comparison with Quotient Trace where we reduce Message Sets that have the same common attribute.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Certain types of blind SQL injection attacks may leave log messages with just one byte difference. We call with analysis pattern Ultrasimilar Messages by analogy with an ultrametric space in mathematics and the interpretation of messages as p-adic numbers. Since, such messages may be scattered in a log we can choose Message Pattern based on some Message Invariant (for example, parts of SQL request) and then analyze its Fiber of Activity (for example, Data Flow of its variable part). A log with two different types of Ultrasimilar Messages is shown in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The advent of virtual machines, the possibility of saving complete memory snapshots without interruption, and the ability to quickly convert such snapshots into a debugger readable memory dump format such as in the case of VMware allows to study how Stack Trace Collections and Wait Chains change over time in complex problem scenarios. Such Stack Trace Surface may also show service restarts if PID changes for processes of interest. We call this pattern by analogy with a memory dump surface where each line corresponds to an individual memory snapshot with coordinates from 0 to the highest address:

In case of orbifold memory space we have a case of a 3D volume (we may call 3D orbifold).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The previous mathematical definition of memory dump is for raw memory dumps. They are not really useful because they require symbol files. Each symbol file entry conceptually is a correspondence between a memory address and a direct sum or product of letters from some alphabet:

00000000`76e82c40: kernel32!WaitForMultipleObjectsExImplementation

So we propose an analytical definition of a memory dump as a direct sum of disjoint memory areas M_t taken during some time interval (t₀, …, t_n) where we replace s_tk having values from Z₂ with S_tq having values from Z_p and cardinality of Z_p depending on a platform (32, 64, etc) plus a symbolic description ∏D_i for each S_tq with cardinality of ”i” set sufficient enough to accommodate the largest symbolic name:

M = ∑M_t where M_t = ∑(S_tq+∏D_i)

or simply

M = ∑∑(S_tq+∏D_i)

This can be visualized as a linear memory space such as a virtual memory space when symbol files are applied to modules one after another. However, all this is not necessary, as a symbol from a virtual address can also be mapped to a physical address if necessary. ∏D_i, in fact, refers to any symbolic description.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

• Title: Memory Dump Analysis Anthology, Volume 1
• Author: Dmitry Vostokov
• Publisher: OpenTask (Summer 2012)
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 800 pages
• ISBN-13: 978-1-908043-35-1

• Hardcover: 800 pages
• ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoretics as a science of memory snapshots borrows many ideas from the following disciplines (the list is not exhaustive):

• Troubleshooting and Debugging
• Intelligence Analysis
• Critical Thinking
• Forensics
• Linguistics
• Archaeology
• Psychoanalysis
• History
• Mathematics: Sets and Categories
• Literary Criticism and Narratology

It also contributes many ideas back. The following diagram depicts such an interaction:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 5 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+5

It features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Its table of contents is available here:

www.dumpanalysis.org/MDAA/MDA-Anthology-V5-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 6 that should be available in November-December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Orbifold Memory Space

A multiple virtual/physical memory space view taking into account multiple computers:

The picture can be much more complex if we glue different manifold memory spaces. The space name comes from a mathematical orbifold, a generalization of manifold.

Synonyms: cloud memory space

Antonyms:

Also: memory space, memory region, physical memory, virtual memory, manifold memory space, memory mapping.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

With the new year starts the new initiative to integrate traditional multidisciplinary debugging approaches and methodologies with multiplatform pattern-driven software problem solving, unified debugging patterns, best practices in memory dump analysis and software tracing, computer security, economics, and the new emerging trends I’m going to write about during this year.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As the new decade is approaching (2011-2020) we would like to make a few previews and predictions:

- Increased complexity of software will bring more methods from biological, social sciences and humanities in addition to existing methods of automated debugging and computer science techniques

- Focus on first fault software problem solving (when aspect)

- Focus on pattern-driven software problem solving (how aspect)

- Fusion of debugging and malware analysis into a unified structural and behavioral pattern framework

- Visual debugging, memory and software trace visualization techniques

- Software maintenance certification

- Focus on domain-driven troubleshooting and debugging tools as a service (debugware TaaS)

- Focus on security issues related to memory dumps and software traces

- New scripting languages and programming language extensions for debugging

- The maturation of the science of memory snapshots and software traces (memoretics)

Imagining is not not limited to the above and more to come and explain in the forthcoming parts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca

The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Product information:

• Title: Memory Dump Analysis Anthology, Volume 5
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-96-4

• Hardcover: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Tool Objects name was motivated by the title of the book Tool and Object: A History and Philosophy of Category Theory that I’m reading:

I realized that troubleshooting can be represented as a category of memory states (or collections of proximate states) as objects and troubleshooting tools as arrows.

We can also consider tools as categories with arrows as troubleshooting actions. In the latter case we can model a collection of tools and transformations (morphisms) between them as a functor.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Now we propose the next group of general patterns related to memory regions (the terminology was partially influenced from topology). The first one we call Memory Region, for example:

1: kd> dp fffff88005875000 fffff88005875010
fffff880`05875000  039b5000`6e696268 00000000`00002000
fffff880`05875010  00000000`00000000

There are Open and Closed memory regions. We can extend the former ones in one or both directions:

1: kd> dp fffff88005875000-10 fffff88005875010+10
fffff880`05874ff0  6ab90c4f`039ba1b8 039b8050`fffffff8
fffff880`05875000  039b5000`6e696268 00000000`00002000
fffff880`05875010  00000000`00000000 00000000`00000000
fffff880`05875020  00000294`ffffffd0

The closed regions cannot be read past its boundary, like this kernel stack region [fffff880`05874000 fffff8800587d000):

1: kd> dp fffff88005874000-30
fffff880`05873fd0  ????????`???????? ????????`????????
fffff880`05873fe0  ????????`???????? ????????`????????
fffff880`05873ff0  ????????`???????? ????????`????????
fffff880`05874000  039ba000`6e696268 00000000`00001000
fffff880`05874010  00000000`00000000 00000000`00000000
fffff880`05874020  00206b6e`ffffffa8 01cae7bd`b8aca323
fffff880`05874030  039b6698`00000000 00000000`00000001
fffff880`05874040  ffffffff`039bafe8 039b6710`00000004

1: kd> dp fffff8800587d000-30
fffff880`0587cfd0  00000000`00000000 00000000`00000000
fffff880`0587cfe0  00000000`00000000 00000000`00000000
fffff880`0587cff0  00000000`00000000 00000000`00000000
fffff880`0587d000  ????????`???????? ????????`????????
fffff880`0587d010  ????????`???????? ????????`????????
fffff880`0587d020  ????????`???????? ????????`????????
fffff880`0587d030  ????????`???????? ????????`????????
fffff880`0587d040  ????????`???????? ????????`????????

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -