Crash Dump Analysis

There are many different approaches to illustrate virtual to physical memory mapping on systems with paging like Windows. Here is another approach that uses natural memory visualization. An image of a user process was generated and juxtaposed to an image of kernel memory dump generated afterwards to produce the combined picture of the full virtual space. Of course, uncommited regions were not included in it as they were not present in user and kernel dumps. Then, after reboot, the same application was launched again and an image of a complete memory dump was generated. Finally, both images were juxtaposed to produce this approximate picture:

In the virtual memory space to the left we see much more granularity. On the contrary, the physical memory space to the right is more uniform and has a different coloring.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

My drive to generalization led me to place an adornment on the portal to highlight the fact that memory and software trace analysis patterns are under an umbrella of general software behaviour patterns:

http://www.dumpanalysis.org/Software-Behavior-Patterns-Headline

In the forthcoming post series I plan to write about similarities between these two branches and also provide pattern examples from non-Windows platforms. All this material will provide the foundation for the forthcoming book Software Behavior: A Guide to Systematic Analysis (ISBN: 978-1906717162).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’ve decided to adapt archaeological classificatory framework (using my favourite method of inquiry: metaphorical bijectionism) to lay out foundations for yet another attempt to classify DA+TA patterns):

Attribute  ↔ Pattern
Artefact   ↔ Component Artefact1
Assemblage ↔ Component Assemblage
Culture    ↔ Memory System Culture²

1 Can be either a component-generated artefact or a component like a module or symbol file
2 Typical examples of memory system cultures are Windows, UNIX or even “Multiplatform”

I propose a word Memoarchaeological for such a framework and Memoarchaeology for a branch of Memoretics that studies saved computer memory artifacts from past computations (as opposed to live memory).

Note: In one of the forthcoming issues of Debugged! MZ/PE magazine there will be presented yet another classificatory scheme.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Let’s now introduce collective pointers or pointer cones. Suppose we have a set of pointers pointing to fields of some memory structure. This set of pointers could be another structure as well or just a collection of pointers that can be logically brought together:

If we make the boundary opaque we can name such set of pointers as Collective Pointer (or Pointer Cone):

Another example is when we split the perception field of a pointer into disjoint collective pointers (the perception field as a whole is already a trivial collective pointer):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This post was motivated during my work on a memory dump differing tool called DumpLogic that can do logical and arithmetic operations between memory snapshots, for example, take a difference between them for further visualization. This tool is forthcoming next week and it resulted in another simple tool called DumpFilter. The latter allows to filter certain unsigned integer (DWORD) values from a memory dump (or any binary file) by replacing them with 0xFFFFFFFF and all other values with 0×00000000. The resultant binary file can be visualized by any data visualization package or transformed to a bitmap file using Dump2Picture to see distribution of filtered values.

As a filtering example I used TestDefaultDebugger64 to generate a process user memory dump. It was converted to a BMP file by Dump2Picture: 

Then I filtered only AV exception code 0xc0000005:

C:\>DumpFilter tdd64.dmp tdd64.bin <dwords.txt

dwords.txt just contained one line 

c0000005

It is possible to filter many values. Just put more lines to dwords.txt file. tdd64.bin was converted to tdd64.bmp by Dump2Picture:

C:\>Dump2Picture tdd64.bin tdd64.bmp

Because the image had only black and while RGBA colors I saved it as a B/W bitmap (click to enlarge, it is a 3236×3236 1.3Mb bitmap):

Every EV exception code is a white dot there but it is difficult to see them unless magnified. So I enlarged them manually on the following map:

I put them on the original image too. We can see that exception processing spans many areas:

The tool and the sample dwords.txt file (for c0000005 and 80000003) can be downloaded from this location:

Download DumpFilter

Another example: Night Sky memory space art image is just a fragment after filtering all 1 values from another process memory dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Computer memory analysis is based on interconnected structures of symbols and we state that there exists a memory language that extends a hierarchy of modeling and implementation languages (both domain-specific and general-purpose):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In part 1 we defined MemP category and the operating field of a pointer as its link to a memory location it is pointing to. This operating field value can be in a different pseudo-memory plane if its value is outside memory bounds, for example, 8FFFFFF0 for a memory with the highest possible address 7FFFFFFF:

We define the closure of memory as the smallest MemP category that includes memory for operating fields of every pointer for the current memory snapshot. For example above, by adding another memory location that has a pointer value pointing back to the original memory region we have the following operating closure:

We can also add more memory as well:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I started applying category theory (as an alternative to traditional set-theoretic approach of memory bits) to memory dump analysis, debugging and software trace analysis in parallel to my studies of that branch of mathematics and reading the book Memory Evolutive Systems. In addition to complex systems modelled in the latter book I apply evolutive systems approach to computer memory. Here is a picture illustrating MemD category of memory dumps (snapshots) as category objects and category arrows as different ways in arriving at the same memory picture:

This category definitely applies to software traces as well if we consider every individual trace message or statement as a minidump. We currently consider software trace category MemT as a subcategory of MemD.

Configuration category of a computer memory dump represents its memory internals at an instant t (ideal memory dumps) or at a time interval T: components and links, pointers, wait chains, causal relations, data flows, … .

Pointers and their links are also objects and arrows to form a category, called MemP(tr). The following picture illustrates it with the last pointer shown as a dereference fixpoint:

The perception field of a pointer is a category of all links to its memory location:

However, the operating field of a pointer is its link to a memory location it is pointing to.

- Dmitry Vostokov @ DumpAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday. The SMI web site temporary points to Crash Dump Analysis and Debugging Portal: www.sminstitute.com

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

Finally, after the long delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Software Tracing

- Dmitry Vostokov @ DumpAnalysis.org -

A nibble is a (0,1)-matrix, a byte is a cubic 0,1-lattice and the next cubic byte-boundary 0,1-lattice represents a 64-bit qword:

This is what I call a natural memory representation as memory building blocks or qubic memory (do not mistaken it with qubit memory). This elevates bytes and 64-bit quadruple words as natural addresses and shows that 32-bit addresses are unnatural.

This also allows to us to visualize certain overlapped memory patterns in dump files (same vertice, edge or side).

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

Consider this example mapping (taken metaphorically from the mathematical notion of an injection) of one domain of knowledge to another:

This mapping between concepts and ideas was once called “bijectivism” but was trivially described either as one to one mapping between two domains (like physical vs. mathematical) or fusing different concepts together to get another emerging concept. I myself proposed the similar mapping and called it a metaphorical bijection.  

Now consider another mapping metaphorically equivalent to a mathematical notion of a surjection where all constituents of the second domain are covered metaphorically by the first domain:

What we strive for is to establish the complete bijective mapping and reorganize our knowledge of both domains to achieve that:

In diagrams above small boxes can represent sets of ideas, methods, etc. or individual ideas, methods, etc. The established metaphorical bijection can divide sets or combine them if needed. There can be several such bijections, of course, and we can use other methods of inquiry (for example, the scientific method) to choose between competing metaphorical bijections.

Useful mnemonic:

BEIS (B=I+S or to BE IS …)

Bijectionism Equals Injection + Surjection

Another mnemonic:

BET (B=T or to BE Transformation…)

Bijectionism Equals Transformation 

Note also the second letter of Alef-Beis or Alef-Bet, the letter of Light that has interpretation of Creation in Biblical Hebrew.   

More on this later as I need to come back to DebugWare patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

Quantum computation, quantum memory and quantum information are hot topics today. Unfortunately quantum mechanics forbids perfect (ideal) memory dumps due to the so called no cloning theorem. Still it is possible to get inconsistent (imperfect) memory dumps and perfect memory dumps can be made from quantum computer simulators. The analysis of quantum memory snapshots is the domain of Quantum Memoretics.

- Dmitry Vostokov @ DumpAnalysis.org -

I’ve been trying to put memory dump analysis on relevant scientific grounds for some time and now this branch of science needs its own name. After considering different alternative names I finally chose the word Memoretics. Here is the brief definition:

Computer Memoretics studies computer memory snapshots and their evolution in time.

Obviously this domain of research has many links with application and system debugging. However its scope is wider than debugging because it doesn’t necessarily study memory snapshots from systems and applications experiencing faulty behaviour.

Initially I was thinking about Memogenics word but its suffix is heavily associated with genes metaphor which I’m currently trying to avoid although I personally re-discovered software genes approach to software disorders when thinking about Memoretics vs. Memogenics. Later I found some research efforts going on but seems they are based on constructing software genes artificially. On the contrary I would try to discover genes in computer memories first.

genic

Also Memoretics has longer prefix almost resembling Memory word. This had the final influence on my decision.

PS. I was also thinking about Memorology word but it has negative connotations with Astrology or Numerology and was coined already by someone like Memology and Memorics. 

- Dmitry Vostokov @ DumpAnalysis.org -