Software Diagnostics Library

Motivated by The Science of Sherlock Holmes I plan to write a book about the history of debugging with the following preliminary product details and tentative release date (which may come earlier if I have enough time):

• Title: The Science of Dr. Watson: An Illustrated History of Debugging
• Author: Dmitry Vostokov
• Publisher: Opentask (01 September 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-07-0
• Paperback: 256 pages

- Dmitry Vostokov @ DumpAnalysis.org -

Following the introductory division of memory analysis into two broad categories I decided to plan yet another book with the following title and preliminary product details:

• Title: Memory Analysis Forensics and Intelligence: An Integral Approach
• Author: Dmitry Vostokov
• Publisher: Opentask (01 September 2009)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-05-6
• Paperback: 256 pages

As you might have noticed, I prefer to put 128, 256 or 512 pages in my book announcements. What that would say about the author’s background? I would like to set 1024 pages for my memory dump analysis anthology series but 740 or 800 pages is the limit for POD technology I use.

- Dmitry Vostokov @ DumpAnalysis.org -

Recently being interested in forensic science and intelligence I decided to elaborate on memory analysis classification and came up with the following division:

- Memory Analysis Forensics

Answering questions related to a committed computer crime. The suspect may be a human or a software / hardware component. Incident response, troubleshooting and debugging belong to this category. Postmortem memory analysis is usually analysis of dump files saved and detached from the original system or operating conditions.

- Memory Analysis Intelligence

Monitoring memory state for behavioural and structural patterns to prevent certain events from occurring. Usually done in situ. However digital dumpster divers and spies may also collect and analyze memory data that was detached from the original computer system.

Each category can be further subdivided into:

– Functional Memory Analysis

Tracing of events.

– Memoretics 

Analysis of memory states and their evolution.

The latter can be subdivided into: 

— Static Memory Analysis

Traditional memory dump analysis.

— Dynamic Memory Analysis

Live debugging.

- Dmitry Vostokov @ DumpAnalysis.org -

After Bug Check Frequencies post I was curious enough to do the same Google counting procedure for exceptions. Here are my results for exceptions listed in Visual C++ Debug \ Exceptions dialog:

The corresponding graph:

- Dmitry Vostokov @ DumpAnalysis.org -

Due to demand from people that prefer ebooks I published Memory Dump Analysis Anthology, Volume 1 in a digital format that can be purchased in Crash Dump Analysis Store. This format has color pictures inside.

- Dmitry Vostokov @ DumpAnalysis.org -

Similar to radiometric dating using isotopes we can use memory visualization techniques to see distribution of allocated buffers and their retention over time. The key is to allocate colored memory. For example, to append a red buffer that contains RGBA values 0xFF000000 to specific allocations. I call these colored memory marks isomemotopes. 

We can either inject a different isomemotope for a different data or change the isomemotope over time to mark specific allocation times. I created a test program that allocates buffers marked by a different amount of different isomemotopes every time: 

#include "stdafx.h"
#include <stdlib.h>
#include <memory.h>
#include <windows.h>

typedef unsigned int ISOMEMOTOPE;

void *alloc_and_mark_with_isomemotope(size_t size,
                                     ISOMEMOTOPE color,
                                     size_t amount)
{
  char *p = (char *)malloc(size+amount);

  for (char *isop = p+size;
       p && isop  < p+size+amount;
       isop+=sizeof(ISOMEMOTOPE))
  {
    *(ISOMEMOTOPE *)isop=color;
  }

  return p;
}

int _tmain(int argc, _TCHAR* argv[])
{
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFF000000, // red 
                                 0x10000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0x00FF0000, // green
                                 0x20000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0x0000FF00, // blue
                                 0x30000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFFFFFF00, // white
                                 0x40000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFFFF0000, // yellow
                                 0x50000);

  DebugBreak();

  return 0;
}

Corresponding Dump2Picture image is this (0×00000000 address is at the bottom):

- Dmitry Vostokov @ DumpAnalysis.org -

I’m very proud to announce that it is finally available in both paperback and hardback. Why have I made available both editions? Because I personally prefer hardcover books. You can order the book today and it will be printed in 3-5 days (paperback) or 5-10 days (hardcover) and sent to you:

Memory Dump Analysis Anthology, Volume 1

Note: although listed on Amazon and other online bookstores it is not immediately available at these stores at the moment due to the late submission. I apologize for this. However, I expect that in a few weeks pre-orders taken there will be eventually fulfilled. In the mean time, if you want the book now, you can use the link above.

- Dmitry Vostokov @ DumpAnalysis.org -

Although the first volume has not been published yet (scheduled for 15th of April, 2008) the planning for the second volume has already begun. Preliminary information is:

• Title: Memory Dump Analysis Anthology, Volume 2
• Paperback: 512 pages (*)
• ISBN-13: 978-0-9558328-7-1
• Author: Dmitry Vostokov
• Publisher: Opentask (01 Oct 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

Hardcover version is also planned. PDF version will be available for download too.

(*) subject to change

- Dmitry Vostokov @ DumpAnalysis.org -

259 bugchecks are documented in WinDbg help. I did Google search for every one and here is their search results distribution graph cut off for data with less than 10 matches:

Of course there is some noise and matches do not always correspond to WinDbg bugcheck analysis output but we can get rough idea about bugcheck frequency. For example, unhandled exceptions in kernel mode, IRQL contract violation, pool corruption and hardware failures are the most frequent. Here is the full table:

- Dmitry Vostokov @ DumpAnalysis.org -

It is very easy to become a publisher nowadays. Much easier than I thought. I registered myself as a publisher under the name of OpenTask which is my registered business name in Ireland. I also got the list of ISBN numbers and therefore can announce product details for the first volume of Memory Dump Analysis Anthology series:

Memory Dump Analysis Anthology, Volume 1

• Paperback: 720 pages (*)
• ISBN-13: 978-0-9558328-0-2

• Hardcover: 720 pages (*)
• ISBN-13: 978-0-9558328-1-9

• Author: Dmitry Vostokov
• Publisher: Opentask (15 Apr 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

(*) subject to change 

PDF file will be available for download too.

- Dmitry Vostokov @ DumpAnalysis.org -

I’ve been trying to put memory dump analysis on relevant scientific grounds for some time and now this branch of science needs its own name. After considering different alternative names I finally chose the word Memoretics. Here is the brief definition:

Computer Memoretics studies computer memory snapshots and their evolution in time.

Obviously this domain of research has many links with application and system debugging. However its scope is wider than debugging because it doesn’t necessarily study memory snapshots from systems and applications experiencing faulty behaviour.

Initially I was thinking about Memogenics word but its suffix is heavily associated with genes metaphor which I’m currently trying to avoid although I personally re-discovered software genes approach to software disorders when thinking about Memoretics vs. Memogenics. Later I found some research efforts going on but seems they are based on constructing software genes artificially. On the contrary I would try to discover genes in computer memories first.

genic

Also Memoretics has longer prefix almost resembling Memory word. This had the final influence on my decision.

PS. I was also thinking about Memorology word but it has negative connotations with Astrology or Numerology and was coined already by someone like Memology and Memorics. 

- Dmitry Vostokov @ DumpAnalysis.org -

Recently became interested in causality, causal models and how they can be applied to software troubleshooting especially when we have various traces and logs. Looking at traces, system and application event logs and logs from other tools, technical support engineers see correlations between various events and build causal models that are used to trace symptoms back to their causes. They use prior knowledge, assumptions, informed guessing and event order to discern causal structure. Clearly event order in logs influences that so it is important to understand how we think in causal terms in order to learn about our biases.

Another important question from software engineering perspective is how to design tracing components to help technical support and software maintenance engineers build correct causal models of software issues. Just finished reading a book about causal modeling:

Review of Causal Models

- Dmitry Vostokov @ DumpAnalysis.org -

Software can be considered as models of real or imagined systems which may be models themselves. Any modeling act involves a mapping between a system and a model that preserves causal, ordering and inclusion relationships and a mapping from the model to the system that translates emerging relationships and causal structures back to that system. The latter I call modeling expectations and any observed deviations in structure and behavior between the model and the system I call software defects which can be functional failures, error messages, crashes or hangs (red line on diagrams below):

Consider ATM software as a venerable example. It models imagined world of ATM transactions which we call ATM software requirements. The latter specifies ACID (atomic, consistent, isolated and durable) transaction rules. If they are broken by the written software we have the defect:

What are software requirements? They are models of real or imagined systems or can be models of past causal and relationship experiences. If requirements are wrong they do not translate back and we still consider software as having a defect:

Translating this to ATM example we have:

Another example where the perceived absence of failures can be considered as a defect is the program designed to model memory leaks that might not be leaking due to a defect in its source code.

Now we can answer the question ”What is a memory corruption?” which is the topic of the next post.

- Dmitry Vostokov @ DumpAnalysis.org -

Asking right questions at the appropriate hierarchical organization level is a known solution to complexity. In case of memory dumps it is sometimes useful to forget about bits, bytes, words, dwords and qwords, memory addresses, pointers, runtime structures, API and ask educated questions at component level, the simplest of it is the question about component timestamp, in WinDbg parlance, using variants of lm command, for example:

0:008> lmt m ModuleA
start    end        module name
76290000 762ad000   ModuleA  Sat Feb 17 13:59:59 2007 (45D70A5F)

0:008> lmt m ModuleB
start    end        module name
66c50000 66c65000   ModuleB  Fri Feb 02 22:30:03 2007 (45C3BB6B)

The next step is obvious: test with the newer version. Another good question is about consistency to exclude cases caused by α-particle hits. This latter possibility was mentioned in Andreas Zeller’s book I read some time ago and can be considered as the efficient cause of some crash dumps according to Aristotelian causation categories.   

- Dmitry Vostokov @ DumpAnalysis.org -

Obviously the appearance of crash dumps on your computer was caused by something. A bug, fault, defect or something else?

Aristotle suggested 4 types of causation 2 millennia ago and they are:

Material cause - presence of some substance, usually material one (hardware) but can be machine code (software). The distinction between hardware and software is often blurred today because of virtualization.

Formal cause - some form or arrangement (an algorithm)

Efficient cause - an agent (data flow or event caused an algorithm to be executed)

Final cause - the desire of someone (or something, operating system, for example).

We skip material causes because hardware and software are always involved. Obviously final causality should be among of crash dump causes because they were either anticipated or made deliberately. Let’s look at 3 examples with possible causes:

Buffer Overflow

• Formal cause - a defect in code which might have arisen from incomplete or wrong model

• Efficient cause - data is too big to fit in a buffer

• Final cause - operating system and runtime library support decided to save a crash dump

Bugcheck (NMI)

• Formal cause - NMI handler

• Efficient cause - a button on a hardware panel or KeBugCheckEx

• Final cause - “I need a memory dump” desire. Also crash dump saving functions were written before by kernel developers in anticipation of future crash dumps.

Bugcheck (A)

• Formal cause - a defect in code again or particular disposition of threads

• Efficient cause - Driver Verifier triggered paging out data

• Final cause - deliberate OS bugcheck (here we can also say that it was anticipated by OS designers)

Concrete causes depend on the organizational level you use: software/hardware systems/components, modeling act by humans, etc.

- Dmitry Vostokov @ DumpAnalysis.org -

Component injection can be used to model various process and system software behavior by writing customized DLL/SYS and injecting them into process/kernel space. Although often depicted either as security threat or value-added hooking mechanism very little has been written about its use to model various software defects. Here I don’t mean testing but studying faulty behavior and artifacts after injecting specific DLLs with design and implementation defects. For example, forgetting to release database connections or not closing file handles. NotMyLeak is an attempt to do it for different kind of leaks on x86 and x64 Windows platforms. It uses automatic DLL injection via standard Windows hooking mechanism. Stay tuned.

- Dmitry Vostokov @ DumpAnalysis.org -  

What are these? These are names of the number of possible unique complete memory dumps when address space is 32 bit and 64-bit correspondingly:

256²³² and 256²⁶⁴

The first of them can be approximated by 10¹⁰¹⁰

This idea came to me after I learnt about the so called “immense number” proposed by Walter Elsasser. This number is so big that its digits cannot be listed because there is not enough particles in observable Universe to write them.

Certainly one memorillion is more than one googol 10¹⁰⁰ but it requires only approx. 10¹⁰ particles in ideal case to list its digits and therefore not an immense number. It is however far less than one googolplex 10¹⁰¹⁰⁰.

Consider a complete memory dump with bytes written in hexadecimal notation:

0x50414745554d500f000000ce0e00000090...

This number has more than 8 billion digits… And it is one possible number out of memorillion of them. So one memorillion in hexadecimal notation is just

0xFFFFFFFFFFFFFFFFFFFFF... + 1

where we have 2*2³² ‘F’ symbols written sequentially. One quadrimemorillion has 2*2⁶⁴ ‘F’ symbols.

Also the question about the number of possible crash dumps can be considered as Microsoft interview style question when you have possible candidates and you want to assess their ability to think out of the box and handle large numbers. 

- Dmitry Vostokov @ DumpAnalysis.org -

From a computer system we get a memory dump composed from fixed size observable values called bit or byte values. Then we impose some structure on it in order to extract various derived objects like threads, processes, etc., build some organization and understand what had happened. This activity is called modeling and memory, crash or core dump analysis is all about modeling a dynamical computer system based on its memory slice. Then we can make predictions and test them via controlled experiments called troubleshooting advices. Tools like WinDbg or GDB can be considered as abstract computers whose job is to model another computer when we feed memory dumps to them.

Modeling computers on computers is inherently reductionist approach and most of mainstream science is just plain reductionism. Just compare the notion of point-like particles as building blocks in physics and bits, recursiveness of physical states and computer algorithms.   

If you want to understand reductionist modeling in physics I would recommend the following book The Comprehensible Cosmos: Where Do the Laws of Physics Come From? written by Victor J. Stenger which I have just finished reading.

The nice feature of this book is its clear separation between textual description and mathematics. The first 190 pages don’t have any mathematical formulas and the next 130 pages repeat the same discussion using undergraduate level of mathematics. 

- Dmitry Vostokov @ DumpAnalysis.org -

In the past I was trying to find a way to depict running and blocked threads graphically perhaps as strings in some abstract n-dimensional space (manifold), preferably 3-dimensional manifold. If you have never encountered manifolds here is their informal definition:

3-dimensional manifold is a 3-dimensional space that looks like a 3-dimensional Euclidean space locally (in small regions) so we can explore the manifold space like we do in our 3-dimensional spatial world

Example: the surface of a sphere where small regions look like 2-dimensional rectangles (compare Earth surface and a football field on it)

My infrequent attempts were not satisfactory and only recently after reading the book Towards a Philosophy of Real Mathematics written by David Corfield I’ve found that it might be good to represent threads as n-string braids.

Braids are strings that raise monotonically without reversing their direction. It sounds like an arrow of time during computation. Braid theory is related to knot theory and might be at a good metaphor to explore. To picture thread strings we need to find abstract coordinates for our space. One of axes is obviously time axis and the other is a program counter axis (for example, the value of EIP register). 

Here is a thread running through code sequentially without jumps or loops, acquiring and releasing a spinlock on its way:

Here is another thread looping while trying to acquire a spinlock and finally taking ownership of it and then running through the same code sequentially:

Suppose that both threads contend for the same spinlock and there is a 3rd thread doing the same. Let’s overlay them on one single diagram:

To have a perspective we can add a 3rd dimension - thread number or ID (TID):

Instead of TID axis we can use data address axis (the data address accessed by the current instruction) or have it as a 4th dimension. If we want to differentiate between read and write addresses we can add 5th axis. We will try to do it in the next part.

- Dmitry Vostokov @ DumpAnalysis.org -

This is the first post in Science of Memory Dump Analysis category where I apply philosophy, systems theory, mathematics, physics and computer science ideas. It was inspired after reading Life Itself book written by Robert Rosen where computers are depicted as direct sums of states. As shown in that book, in the case of machines, their synthetic models (direct sums) are equivalent to analytic models (direct product of observables). Taking every single bit as an observable having its values in Z₂ set {0, 1} we can make a definition of an ideal memory dump as a direct product or a direct sum of bits saved instantaneously at the given time:

∏_i s_i = ∑_i s_i

Of course, we can also consider bytes having 8 bits as observables having their values from Z₂₅₆ set, etc.

In our case we can simply rewrite direct sum or product as the list of bits, bytes, words or double words, etc:

(…, s_i-1, s_i, s_i+1, …, s_j-1, s_j, s_j+1, …)

According to Rosen we include hardware states (registers, for example) and partition memory into input, output states for particular computation and other states.

Saving a memory dump takes certain amount of time. Suppose that it takes 3 discrete time events (ticks). During the first tick we save memory up to (…, s_i-1, s_i) and that memory has some relationship to s_j state. During the second tick s_j state changes its value and during the 3rd tick we copy the rest of the memory (s_i+1, …, s_j-1, s_j, s_j+1, …). Now we see that the final memory dump is inconsistent:

(…, s_i-1, s_i, s_i+1, …, s_j-1, s_j, s_j+1, …)

I explained this earlier in plain words in Inconsistent Dump pattern. Therefore we might consider a real memory dump as a direct sum of disjoint memory areas M_t taken during some time interval (t₀, …, t_n)

M = ∑_t M_t where M_t = ∑_k s_tk or simply

M = ∑_t ∑_k s_tk

- Dmitry Vostokov @ DumpAnalysis.org -