Crash Dump Analysis

This post was motivated during my work on a memory dump differing tool called DumpLogic that can do logical and arithmetic operations between memory snapshots, for example, take a difference between them for further visualization. This tool is forthcoming next week and it resulted in another simple tool called DumpFilter. The latter allows to filter certain unsigned integer (DWORD) values from a memory dump (or any binary file) by replacing them with 0xFFFFFFFF and all other values with 0×00000000. The resultant binary file can be visualized by any data visualization package or transformed to a bitmap file using Dump2Picture to see distribution of filtered values.

As a filtering example I used TestDefaultDebugger64 to generate a process user memory dump. It was converted to a BMP file by Dump2Picture: 

Then I filtered only AV exception code 0xc0000005:

C:\>DumpFilter tdd64.dmp tdd64.bin <dwords.txt

dwords.txt just contained one line 

c0000005

It is possible to filter many values. Just put more lines to dwords.txt file. tdd64.bin was converted to tdd64.bmp by Dump2Picture:

C:\>Dump2Picture tdd64.bin tdd64.bmp

Because the image had only black and while RGBA colors I saved it as a B/W bitmap (click to enlarge, it is a 3236×3236 1.3Mb bitmap):

Every EV exception code is a white dot there but it is difficult to see them unless magnified. So I enlarged them manually on the following map:

I put them on the original image too. We can see that exception processing spans many areas:

The tool and the sample dwords.txt file (for c0000005 and 80000003) can be downloaded from this location:

Download DumpFilter

Another example: Night Sky memory space art image is just a fragment after filtering all 1 values from another process memory dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In part 1 we defined MemP category and the operating field of a pointer as its link to a memory location it is pointing to. This operating field value can be in a different pseudo-memory plane if its value is outside memory bounds, for example, 8FFFFFF0 for a memory with the highest possible address 7FFFFFFF:

We define the closure of memory as the smallest MemP category that includes memory for operating fields of every pointer for the current memory snapshot. For example above, by adding another memory location that has a pointer value pointing back to the original memory region we have the following operating closure:

We can also add more memory as well:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I started applying category theory (as an alternative to traditional set-theoretic approach of memory bits) to memory dump analysis, debugging and software trace analysis in parallel to my studies of that branch of mathematics and reading the book Memory Evolutive Systems. In addition to complex systems modelled in the latter book I apply evolutive systems approach to computer memory. Here is a picture illustrating MemD category of memory dumps (snapshots) as category objects and category arrows as different ways in arriving at the same memory picture:

This category definitely applies to software traces as well if we consider every individual trace message or statement as a minidump. We currently consider software trace category MemT as a subcategory of MemD.

Configuration category of a computer memory dump represents its memory internals at an instant t (ideal memory dumps) or at a time interval T: components and links, pointers, wait chains, causal relations, data flows, … .

Pointers and their links are also objects and arrows to form a category, called MemP(tr). The following picture illustrates it with the last pointer shown as a dereference fixpoint:

The perception field of a pointer is a category of all links to its memory location:

However, the operating field of a pointer is its link to a memory location it is pointing to.

- Dmitry Vostokov @ DumpAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday. The SMI web site temporary points to Crash Dump Analysis and Debugging Portal: www.sminstitute.com

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

Finally, after the long delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Software Tracing

- Dmitry Vostokov @ DumpAnalysis.org -

A nibble is a (0,1)-matrix, a byte is a cubic 0,1-lattice and the next cubic byte-boundary 0,1-lattice represents a 64-bit qword:

This is what I call a natural memory representation as memory building blocks or qubic memory (do not mistaken it with qubit memory). This elevates bytes and 64-bit quadruple words as natural addresses and shows that 32-bit addresses are unnatural.

This also allows to us to visualize certain overlapped memory patterns in dump files (same vertice, edge or side).

- Dmitry Vostokov @ DumpAnalysis.org -

Although everything changes in the philosophy of Heraclitus, its Urstoff, fire, is eternal, indestructable. The same can be said about Memory and its memory fragments. Memoidealism has also some proximity with the notion of everlasting existence of thought of or spoken named objects in the philosophy of Parmenides. Once we think of or speak of something we immediately recall a memory fragment from and, at the same time, commit it to Memory again, perhaps as a different memory fragment, an assembled new memory trace.

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Analysis and Debugging Institute (MA&DI), DA+TA Portal (DumpAnalysis.org + TraceAnalysis.org) and OpenTask establish

Software Maintenance Institute (SMI)

More details about R&D Institute of Software Maintenance will be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

The notion of transmigration of memories in Memoidealism has its similarity with Orphicism. The notions of limited (memories) and unlimited memory (Memory), and “things are memories” similar to Pythagoreanism view on Limited and Unlimited, and “things are numbers”.

- Dmitry Vostokov @ DumpAnalysis.org -

Consider this example mapping (taken metaphorically from the mathematical notion of an injection) of one domain of knowledge to another:

This mapping between concepts and ideas was once called “bijectivism” but was trivially described either as one to one mapping between two domains (like physical vs. mathematical) or fusing different concepts together to get another emerging concept. I myself proposed the similar mapping and called it a metaphorical bijection.  

Now consider another mapping metaphorically equivalent to a mathematical notion of a surjection where all constituents of the second domain are covered metaphorically by the first domain:

What we strive for is to establish the complete bijective mapping and reorganize our knowledge of both domains to achieve that:

In diagrams above small boxes can represent sets of ideas, methods, etc. or individual ideas, methods, etc. The established metaphorical bijection can divide sets or combine them if needed. There can be several such bijections, of course, and we can use other methods of inquiry (for example, the scientific method) to choose between competing metaphorical bijections.

Useful mnemonic:

BEIS (B=I+S or to BE IS …)

Bijectionism Equals Injection + Surjection

Another mnemonic:

BET (B=T or to BE Transformation…)

Bijectionism Equals Transformation 

Note also the second letter of Alef-Beis or Alef-Bet, the letter of Light that has interpretation of Creation in Biblical Hebrew.   

More on this later as I need to come back to DebugWare patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

Last week I was comparing the existing collection of memory dump analysis patterns to the collection of trace analysis patterns (in formation) in the search of isomorphism (or more correctly, general morphism) similar to Missing Component pattern. It is not a coincidence that such pattern pairs can be formed. For example, it is possible to discern deadlocks from both crash dumps and software traces (if appropriate information is available there). Fundamentally, it is implied by the definition of a software trace as some sort of a memory dump. And we can see traces in memory dumps too, for example, Execution Residue pattern. Because raw stack data resides in stack pages and in contemporary operating systems they are created from zero pages (metaphorically, out of the void) we can say that stack regions of threads are sorted by their creation time, for example, in this process user memory dump:

0:017> !runaway 4
 Elapsed Time
  Thread       Time
   0:49c       0 days 5:16:31.076
   4:4d8       0 days 5:16:30.967
   3:4d0       0 days 5:16:30.967
   2:4cc       0 days 5:16:30.967
   1:4c8       0 days 5:16:30.967
   5:4e8       0 days 5:16:30.936
   6:b6c       0 days 5:16:15.695
   7:b70       0 days 5:16:15.679
   9:b88       0 days 5:16:15.586
   8:b84       0 days 5:16:15.586
  11:348       0 days 5:16:12.934
  10:bfc       0 days 5:16:12.934
  12:1200      0 days 5:15:16.528
  15:1298      0 days 5:15:15.220
  14:1290      0 days 5:15:15.220
  13:128c      0 days 5:15:15.220
  17:12e4      0 days 5:15:13.257
  16:12dc      0 days 5:15:13.257
  18:12ec      0 days 5:15:13.117
  20:12f4      0 days 5:15:13.085
  19:12f0      0 days 5:15:13.085
  21:17a0      0 days 5:13:16.321
  22:1628      0 days 5:13:15.729
  24:1778      0 days 1:35:50.773
  23:17ec      0 days 1:35:50.773
  25:1570      0 days 1:27:54.190
  26:1724      0 days 1:27:10.151
  27:1490      0 days 0:05:46.732
  28:1950      0 days 0:02:28.153
  29:19b4      0 days 0:00:58.108
  30:177c      0 days 0:00:38.358
  31:1798      0 days 0:00:23.351
  32:1a7c      0 days 0:00:08.343

If we have complete memory dumps we can also account for other processes and their elapsed time. Within stack pages we have partial stack traces but do not have exact timing information between them except for stack frames from the current frozen thread stack trace or, if we are lucky, from a partial stack trace from the past execution. However, the timing between frames from different stacks is undefined and we can only guess it from higher level considerations like semantics of procedure calls and other information.

These considerations and the notion of a poset (partially ordered set) let me thinking about memory dumps as posets. I even created my interpretation of POSET abbreviation for this occasion:

POSET 

Partially Ordered Software Execution Trace   

- Dmitry Vostokov @ DumpAnalysis.org -

The hierarchy of Ψ₁, …, Ψ₈, …, Ψ₁₆, …, Ψ₃₂, …, Ψ₆₄, …, …, …, Ψ_Ψ numbers where the subscript denotes the number of bits a memory address can have, so Ψ₃₂ and Ψ₆₄ are memorillion and quadrimemorillion of memory dumps respectively. We only need to figure out the meaning of Ψ₀ and Ψ_Ψ. Perhaps there is some meaning in Dirac notation here: <Ψ₀|Ψ_Ψ>. More on this later because I have to finish this week the book x64 Windows Debugging: Practical Foundations and write an errata file for the previous x86 version of the book series.

Note: Ψ is an M upside down.

- Dmitry Vostokov @ DumpAnalysis.org -

If memory is the basis of everything we might think that memoidealism is a kind of dual aspect monism or plural aspect monism, where mind, body and perhaps many other aspects are manifestations of one single memory substance. Or perhaps, memory is the same as matter or the realm of ideas (materialism or some sort of idealism), and memoidealism is the kind of monism. Or memory is the same or not the same as mind or matter and can be associated with one side of dualism. Not at all, memory is the aspect or attribute of mind, body and other substances, possibly itself. This can be illustrated on the following picture:

If there are only 2 substances then we have monistic aspect dualism as an extension of monism, for example, the extension of idealism:

or the extension of materialism (memuonic theory, memuonics):

Therefore, Memory is a substance and an aspect (attribute) at the same time. This is the essence of MAP (Monistic Aspect Pluralism).

- Dmitry Vostokov @ DumpAnalysis.org -

What gives rise to intelligence in memory medium? Apparently the drive towards infinity via power set or the so called exponentiation, where patterns, subsets of memory, are combined in their totality to form even bigger memory space. Imagine how many subsets can be formed from just one complete 4Gb memory dump?

We have 2³² unique byte pairs (address, value)

so we have 2²³² possible subsets.

- Dmitry Vostokov @ DumpAnalysis.org -

When complete memory dumps are huge (in case of x64 systems) we can dump specific processes and then force a kernel memory dump. Here we have a product of spaces similar to a fiber bundle illustrated by the following intuitive picture:

- Dmitry Vostokov @ DumpAnalysis.org -

There are many books and articles titled “Large-scale Structure of X”, where X can be Space-Time, Cosmos or Universe. Here is the large-scale structure of 12Gb complete memory dump:

The image was generated with the help of ImageMagick. The dump file was interpreted as a raw RGBA image with 8-bits per color:

C:\MemoryDumps>convert -size 56751x56751 -depth 8
-resize 450x450 rgba:complete.dmp dump_12Gb_rgba_8_sq.jpg

The width and height were calculated as sqrt(filesize/4).

Complete memory dumps are physical memory dumps where modularized structure of virtual space of kernel and process memory is not expected but we see some structure nevertheless.

I’ve also created two pages with dump slices. Some viewers do not handle files with more that 32767 pixels in one dimension so I split 450×56751 slice into two:

Complete Memory Dump Slice Part 1 (11Mb JPEG)

Complete Memory Dump Slice Part 2 (10Mb JPEG)  

- Dmitry Vostokov @ DumpAnalysis.org -

Following the meaning of DATA and memory dump world view via universal memory dumps I finally deciphered the acronym DUMP:

Digital Universal Memory Phase

This is the view from phase space perspective. From the point of phase space perspective we can also say:

Digital Universal Memory Point

It was the letter P that I was thinking hard about. Fortunately, when I opened Oxford Advanced Learner’s Dictionary on P section, it was “phase” word that grabbed my attention. Familiarity with classical physics and its Hamiltonian formulation provided the necessary glue.

- Dmitry Vostokov @ DumpAnalysis.org -