Software Diagnostics Library

When I was a child I experienced dreams where I was carried by a huge wave that was transforming to a torus completely absorbing me up to a breakpoint of my wake up. A year ago I got the book Memory Evolutive Systems because of my interest in applying category theory to memory analysis and debugging and immediately recalled my long-time forgotten childhood dreams while staring at its front cover:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There are many different approaches to illustrate virtual to physical memory mapping on systems with paging like Windows. Here is another approach that uses natural memory visualization. An image of a user process was generated and juxtaposed to an image of kernel memory dump generated afterwards to produce the combined picture of the full virtual space. Of course, uncommited regions were not included in it as they were not present in user and kernel dumps. Then, after reboot, the same application was launched again and an image of a complete memory dump was generated. Finally, both images were juxtaposed to produce this approximate picture:

In the virtual memory space to the left we see much more granularity. On the contrary, the physical memory space to the right is more uniform and has a different coloring.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

… science is in reality a classification and analysis of the contents of the memory;

Karl Pearson, The Grammar of Science

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

My drive to generalization led me to place an adornment on the portal to highlight the fact that memory and software trace analysis patterns are under an umbrella of general software behaviour patterns:

http://www.dumpanalysis.org/Software-Behavior-Patterns-Headline

In the forthcoming post series I plan to write about similarities between these two branches and also provide pattern examples from non-Windows platforms. All this material will provide the foundation for the forthcoming book Software Behavior: A Guide to Systematic Analysis (ISBN: 978-1906717162).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’ve decided to adapt archaeological classificatory framework (using my favourite method of inquiry: metaphorical bijectionism) to lay out foundations for yet another attempt to classify DA+TA patterns):

Attribute  ↔ Pattern
Artefact   ↔ Component Artefact1
Assemblage ↔ Component Assemblage
Culture    ↔ Memory System Culture²

1 Can be either a component-generated artefact or a component like a module or symbol file
2 Typical examples of memory system cultures are Windows, UNIX or even “Multiplatform”

I propose a word Memoarchaeological for such a framework and Memoarchaeology for a branch of Memoretics that studies saved computer memory artifacts from past computations (as opposed to live memory).

Note: In one of the forthcoming issues of Debugged! MZ/PE magazine there will be presented yet another classificatory scheme.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming CARE and STARE online systems additionally aim to provide software behaviour pattern identification via debugger log and trace analysis and suggest possible software troubleshooting patterns. The purpose of these post series is to provide high level overview of possible patterns of software behavior and how they can be recognised and analyzed. This work started in October, 2006 with the identification of computer memory patterns and later continued with software trace patterns. Bringing all of them under a unified linked framework seems quite natural to me.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Let’s now introduce collective pointers or pointer cones. Suppose we have a set of pointers pointing to fields of some memory structure. This set of pointers could be another structure as well or just a collection of pointers that can be logically brought together:

If we make the boundary opaque we can name such set of pointers as Collective Pointer (or Pointer Cone):

Another example is when we split the perception field of a pointer into disjoint collective pointers (the perception field as a whole is already a trivial collective pointer):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This post was motivated during my work on a memory dump differing tool called DumpLogic that can do logical and arithmetic operations between memory snapshots, for example, take a difference between them for further visualization. This tool is forthcoming next week and it resulted in another simple tool called DumpFilter. The latter allows to filter certain unsigned integer (DWORD) values from a memory dump (or any binary file) by replacing them with 0xFFFFFFFF and all other values with 0×00000000. The resultant binary file can be visualized by any data visualization package or transformed to a bitmap file using Dump2Picture to see distribution of filtered values.

As a filtering example I used TestDefaultDebugger64 to generate a process user memory dump. It was converted to a BMP file by Dump2Picture: 

Then I filtered only AV exception code 0xc0000005:

C:\>DumpFilter tdd64.dmp tdd64.bin <dwords.txt

dwords.txt just contained one line 

c0000005

It is possible to filter many values. Just put more lines to dwords.txt file. tdd64.bin was converted to tdd64.bmp by Dump2Picture:

C:\>Dump2Picture tdd64.bin tdd64.bmp

Because the image had only black and while RGBA colors I saved it as a B/W bitmap (click to enlarge, it is a 3236×3236 1.3Mb bitmap):

Every EV exception code is a white dot there but it is difficult to see them unless magnified. So I enlarged them manually on the following map:

I put them on the original image too. We can see that exception processing spans many areas:

The tool and the sample dwords.txt file (for c0000005 and 80000003) can be downloaded from this location:

Download DumpFilter

Another example: Night Sky memory space art image is just a fragment after filtering all 1 values from another process memory dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In part 1 we defined MemP category and the operating field of a pointer as its link to a memory location it is pointing to. This operating field value can be in a different pseudo-memory plane if its value is outside memory bounds, for example, 8FFFFFF0 for a memory with the highest possible address 7FFFFFFF:

We define the closure of memory as the smallest MemP category that includes memory for operating fields of every pointer for the current memory snapshot. For example above, by adding another memory location that has a pointer value pointing back to the original memory region we have the following operating closure:

We can also add more memory as well:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I started applying category theory (as an alternative to traditional set-theoretic approach of memory bits) to memory dump analysis, debugging and software trace analysis in parallel to my studies of that branch of mathematics and reading the book Memory Evolutive Systems. In addition to complex systems modelled in the latter book I apply evolutive systems approach to computer memory. Here is a picture illustrating MemD category of memory dumps (snapshots) as category objects and category arrows as different ways in arriving at the same memory picture:

This category definitely applies to software traces as well if we consider every individual trace message or statement as a minidump. We currently consider software trace category MemT as a subcategory of MemD.

Configuration category of a computer memory dump represents its memory internals at an instant t (ideal memory dumps) or at a time interval T: components and links, pointers, wait chains, causal relations, data flows, … .

Pointers and their links are also objects and arrows to form a category, called MemP(tr). The following picture illustrates it with the last pointer shown as a dereference fixpoint:

The perception field of a pointer is a category of all links to its memory location:

However, the operating field of a pointer is its link to a memory location it is pointing to.

- Dmitry Vostokov @ DumpAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday.

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

Finally, after the long delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Software Tracing

- Dmitry Vostokov @ DumpAnalysis.org -

A nibble is a (0,1)-matrix, a byte is a cubic 0,1-lattice and the next cubic byte-boundary 0,1-lattice represents a 64-bit qword:

This is what I call a natural memory representation as memory building blocks or qubic memory (do not mistaken it with qubit memory). This elevates bytes and 64-bit quadruple words as natural addresses and shows that 32-bit addresses are unnatural.

This also allows to us to visualize certain overlapped memory patterns in dump files (same vertice, edge or side).

- Dmitry Vostokov @ DumpAnalysis.org -

Although everything changes in the philosophy of Heraclitus, its Urstoff, fire, is eternal, indestructable. The same can be said about Memory and its memory fragments. Memoidealism has also some proximity with the notion of everlasting existence of thought of or spoken named objects in the philosophy of Parmenides. Once we think of or speak of something we immediately recall a memory fragment from and, at the same time, commit it to Memory again, perhaps as a different memory fragment, an assembled new memory trace.

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Analysis and Debugging Institute (MA&DI), DA+TA Portal (DumpAnalysis.org + TraceAnalysis.org) and OpenTask establish

Software Maintenance Institute (SMI)

More details about R&D Institute of Software Maintenance will be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -