Crash Dump Analysis

Today we introduce an icon for Manual Dump (process) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This forthcoming full color book is the complete transcript of a Webinar organized by Memory Dump Analysis Services (www.DumpAnalysis.com).

It discusses user vs. kernel vs. physical (complete) memory space, challenges of complete memory dump analysis, common WinDbg commands, patterns and pattern-driven analysis methodology, common mistakes, fiber bundles, DumpAnalysis.org case studies and illustrates step by step a hands-on exercise in a complete memory dump analysis.

• Title: Fundamentals of Complete Crash and Hang Memory Dump Analysis
• Author: Dmitry Vostokov
• Publisher: OpenTask (October 2010)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 48 pages
• ISBN-13: 978-1906717155

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Memorianic supper: Judaism: The Basics, Derrida (Routledge Critical Thinkers), Irish Freedom, Europe at War, Keywords, Greece and Rome
• Published a horoscope on Software for September: http://softwareastrology.com/2010/09/02/horoscope-on-software-september-2010/
• Was reading articles on agent causation, anglophone African philosophy and account settlement while commuting to the office

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Manual Dump (kernel) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Memorianic supper: Semiotics: The Basics, Martin Heidegger (Routledge Critical Thinkers), Keywords, Software Studies, General Chemistry
• Software trace analysis while listening to some instrumental music
• Memorianic lunch: reading Quantum Generations, Beria, Ayn Rand: The Russian Radical, An Introduction to General Systems Thinking, The Writer
• Memory dump analysis while listening to Beethoven: Vocal Works
• Reading articles on the history of psychology as a social science, general African philosophy and accounting software while commuting home

If you attended Fundamentals of Complete Crash and Hang Memory Dump Analysis you probably remember the memory dump visualization question that I repeat here on this slide fragment:

I got a few responses:

“Unfortunately they are not identical - visual inspection shows that. I tried differencing the relevant sub-images in Photoshop and I can’t get zero.  Of course this can be due to compression artifacts and, more likely, the fact that the duplication is not required to be aligned to the borders. A stronger confirmation/refutation would require unrolling the bitmap to one dimension and sliding it back and forth until maximum correlation is found. Since I have not done the examples step by step, I am left guessing about just what the dump you show illustrates. An aliased memory mapped area is my first guess, and a flip/flop garbage collector is my second.”

“perhaps some module such as a .NET assembly is getting loaded twice in a .NET app, pre .NET 4, such as is dicsussed in this thread:
http://forum.sysinternals.com/why-some-net-assemblies-are-duplicated-in-memory_topic15279_post121591.html“

Initially I also thought that there was the same module loaded twice from different location like in Duplicated Module pattern. Unfortunately lm command didn’t show any duplicated loaded and unloaded modules as well as any hidden modules. So I looked at address information and found two identical relatively large regions at the beginning:

0:000> !address
[...]
BaseAddress EndAddress+1 RegionSize Type       State      Protect       Usage
[...]
0`00470000  0`007f0000   0`00380000 MEM_MAPPED MEM_COMMIT PAGE_READONLY <unclassified>
[…]
0`01f10000  0`02290000   0`00380000 MEM_MAPPED MEM_COMMIT PAGE_READONLY <unclassified>
[…]

The image above was scaled by ImageMagic from a bitmap generated by Dump2Picture:

The original image from Dump2Picture had different colors:

I quickly checked the colorimetric structure of those regions: 0`00470000  0`007f0000 and 0`01f10000  0`02290000 using MemPicture WinDbg script and they seem to conform with the magnified picture above:

0:000> $$>a< d:\Dump2Picture\mempicture.txt 0`00470000 L?0`007f0000-0`00470000
Writing 380000 bytes

C:\Program Files\Debugging Tools for Windows (x64)>dump2picture d2p-range.bin d2p-range.bmp

Dump2Picture version 1.1
Written by Dmitry Vostokov, 2007

d2p-range.bmp
d2p-range.bin
        1 file(s) copied.

C:\Program Files\Debugging Tools for Windows (x64)>d2p-range.bmp
.shell: Process exited

Here is the magnified slice from the original picture:

We see the same partitioning if we juxtapose the original picture and the picture of the address region:

Also these regions are completely identical if we compare their data:

0:000> c 0`00470000 L?(0`007f0000-0`00470000)/8 0`01f10000

So it looks like some file was mapped twice. Inspected via dc command it shows remarkable regularity not seen in executable modules. This regularity also manifests itself in color:

In order to verify I modeled this by writing a simple program that maps a file twice passed as a command line parameter:

int _tmain(int argc, _TCHAR* argv[])
{
 if (argc < 2)
 {
  puts("Usage: MappedFiles.exe <File_Name_To_Map>\n");
  return -1;
 }

 HANDLE hf = CreateFile(argv[1], GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
 HANDLE hm = CreateFileMapping(hf, NULL, PAGE_READONLY, 0, 0, NULL);
 MapViewOfFile(hm, FILE_MAP_READ, 0, 0, 0);

 hf = CreateFile(argv[1], GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
 hm = CreateFileMapping(hf, NULL, PAGE_READONLY, 0, 0, NULL);
 MapViewOfFile(hm, FILE_MAP_READ, 0, 0, 0);

 DebugBreak();
 return 0;
}

I ran it and chose to map explorer.exe because it was a sufficiently large image file:

C:\MappedFiles\Release>MappedFiles.exe c:\windows\explorer.exe

The dump file was saved and its processing shows this picture:

We clearly see identical regions and double check them from the dump file:

0:000> !address
BaseAddr EndAddr+1 RgnSize Type       State         Protect       Usage
[...]
a60000   d1d000    2bd000  MEM_MAPPED MEM_COMMIT    PAGE_READONLY <unclassified>
d1d000   d20000    3000    MEM_FREE   PAGE_NOACCESS Free
d20000   fdd000    2bd000  MEM_MAPPED MEM_COMMIT    PAGE_READONLY <unclassified>
[…] 

0:000> $$>a< d:\Dump2Picture\mempicture.txt 0`00470000 L?0`007f0000-0`00470000
Writing 380000 bytes

C:\Program Files\Debugging Tools for Windows (x64)>dump2picture d2p-range.bin d2p-range.bmp 

Dump2Picture version 1.1
Written by Dmitry Vostokov, 2007

d2p-range.bmp
d2p-range.bin
        1 file(s) copied.

C:\Program Files\Debugging Tools for Windows (x64)>d2p-range.bmp
.shell: Process exited

We see the same partitioning if we juxtapose results:

The application can be downloaded from here: MappedFiles.zip

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Memory dump analysis while listening to Yuichi Watanabe: Piano Hommage #
• Memory dump analysis while listening to Handel: Fireworks and Water Music #
• Code construction while listening to Jethro Tull: Heavy Horses #
• Memorianic lunch: reading The Enemy Within, The Murder of Nikolai Vavilov, A History of God, Sacred Causes, Ayn Rand: The Russian Radical #
• Memory dump and debugger log analysis while listening to Ambient: Around The World compilation album #

Finally I compiled a Questions and Answers page with all necessary links and examples during the weekend:

http://www.dumpanalysis.com/FCMDA-Q-A

I also added text versions of logs (in addition to zip files) to a Webinar materials page:

http://www.dumpanalysis.com/FCMDA-materials

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Memory dump analysis while listening to Deep Forest: Boheme to give some speed boost #
• Memory dump analysis while listening to Jethro Tull: Aqualung #
• Adjoint thread analysis while listening to Mozart: German Dances and Menuets #
• Memorianic lunch: reading Trotsky (Service), The Origin of Consciousness in the Breakdown of the Bicameral Man, A History of God, IBPA mag. #
• Software trace analysis while listening to Bach: Chorale Preludes #
• Was reading articles on statistics in social sciences and African philosophy while commuting to the office #

Just noticed this little book and ordered it to browse and write a review later Glitch: Designing Imperfection

Notice the patterns with glitches on the cover similar to the fabric of memory and some pictures from inside remind me natural memory visualization images you can find in the print form in DLL List Landscape: The Art from Computer Memory Space book. I would therefore welcome the fact that the memory space art has more and more followers (and becomes the true art movement according to the institutional theory of art). There are more art books coming later this year:

Also the museum of debugging and memory dumps will hold an exhibition soon.

In a few days the following book about glitches (as negative business impact annoyances) will be released:

Glitch: The Hidden Impact of Faulty Software

“Search inside the book” on Amazon suggests that the latter book doesn’t consider the positive impact of glitches including art but, anyway, because of my speciality in software debugging and maintenance I would like to review that book too.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Software trace and memory dump analysis while listening to Bach: Orgelbüchlein #
• Late short memorianic lunch: reading books Russia in Search of Itself, An Introduction to General Systems Thinking, A History of God #
• Memory dump and software trace analysis while listening to selected Bach Toccatas, Preludes and Fugues #
• Was reading articles on changing contours of social science disciplines, affirmative action, accounting information systems while commuting #

Today we introduce an icon for Special Stack Trace pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was inspecting the list of timers in a complete memory dump and found that in x64 W2K8 R2 (it may be also in other versions) there is a timer set to fire for the end of our century 2001 - 2100:

1: kd> !timer
Dump system timers
[...]
    fffff80001c31b80   f690c4d5 0064364d [12/31/2099 23:00:00.510]  nt!ExpCenturyDpcRoutine (DPC @ fffff80001c31bc0)
[…]

Its disassembly shows at that time the following work item will be executed:

1: kd> uf nt!ExpCenturyDpcRoutine
[...] 
fffff800`01ae81ab 488d0dae991400  lea     rcx,[nt!ExpCenturyWorkItem (fffff800`01c31b60)]
[...]

I don’t want to dig myself far into the details here May be someone from MS provides some comments what we should expect at the turn of the century? What should I tell my children to expect if they still run the Windows OS?

PS. Long live the Windows OS! (my the most sincere comment; no pun intended).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Like in a literature (and in a narratology in general) we have components that trace themselves and components that tell the story of a computation including status updates they query about other components and subsystems. This pattern gets its name Diegetic Messages from diegesis. Here’s the difference between diegetic (in blue bold) and non-diegetic trace messages:

PID    TID    TIME          MESSAGE     
11864  11912  06:34:53.598  ModuleA: foo called bar. Status OK.
11620  10372  06:34:59.754  ModuleB: ModuleA integrity check. Status OK.

Some modules may emit messages that tell about their status but from their message text we know the larger computation story like in a process startup sequence example.  

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Back from 3 day holiday. Was reading books Trotsky (Service) and Being Geek while at airports and on planes. #

The presentation materials from the webinar (18th and 23rd of August, 2010) are available for download:

http://www.dumpanalysis.com/FCMDA-materials

Thanks to everyone who registered and attended!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Short break to read books: Routledge Encyclopedia of Narrative Theory and Hermeneutics #

I’m pleased to announce that my book Memory Dump Analysis Anthology, Volume 2 is available for Safari Books Online subscribers:

http://my.safaribooksonline.com/9780955832871

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Debugging Experts Magazine Online (online version of Debugged! MZ/PE) underwent the total redesign:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

• Working on a complete redesign of Debugging Experts Magazine Online (Debugged! MZ/PE Online) #
• Was reading an article on problems and theories of aesthetics while commuting today #