Crash Dump Analysis

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday. The SMI web site temporary points to Crash Dump Analysis and Debugging Portal: www.sminstitute.com

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

What prompted me to found this discipline (that is supposed to be a sister discipline of software criminology, software security, secure code construction and software defect construction) is understanding that some software components are innocent victims of other component coding mistakes or deliberate subversion and some start as a part of crimeware but eventually become victims themselves (they crash, hang, spike, leak, are dumped, subverted, etc.). I would also like to borrow and reuse the neglected term victimware here in a broad sense. More on this later as I have to switch to software trace analysis patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org accepts hardware such as laptops for reviewing in relation to their suitability for extreme debugging, virtualization, trace analysis, computer forensics, memory dump analysis, visualization and auralization. If you work for a H/W company like HP, Apple, Dell, Acer, Sony or any other respectable manufacturer please don’t hesitate to forward this post to your management: it could be your company brand or laptop model that debugging and software technical support community chooses next time of upgrade or for T&D / R&D! H/W reviews will be posted on the main portal page which currently has an audience of more than 200,000 unique visitors per year from more than 30,000 network locations (*).

If your company is interested please don’t hesitate to use this contact form:

http://www.dumpanalysis.org/contact

(*) From Google Analytics report.

- Dmitry Vostokov @ DumpAnalysis.org -

SecretSES (Secret Software Engineering Society) announces Debugging Spy Network of Memory Analysis Forensics and Intelligence Agents (MAFIA):

http://www.secretses.com/

Motivation: seeing various nomadic and settled debugging teams I finally decided to unfold my own net.

- Dmitry Vostokov @ DumpAnalysis.org -

I’m very pleased to announce the free online version of Debugged! MZ/PE magazine under the code name DEMO launched last night:

Debugging Expert Magazine Online (www.DebuggingExpert.com)

- Dmitry Vostokov @ DumpAnalysis.org -

Looking at pattern cooperation studies it is easy to see that some patterns precede others, for example, heap corruption might be blocked by a hard error or a message box and therefore block other threads, creating conditions for another pattern to appear, wait chains. Blocked threads may block other coupled processes creating inter-process wait chains. Successive patterns reach the climax at the end and the system is no longer able to generate any other patterns.

The goal here is to find patterns that most likely happen in succession and another pattern series that are less likely to effect other abnormal conditions. Such pattern sequences can help in troubleshooting and finding root causes.

- Dmitry Vostokov @ DumpAnalysis.org - 

Found today this book while browsing Amazon:

The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System

Intrigued, I searched for its table of contents and found the author’s site:

Book TOC

Looks enough comprehensive so I pre-ordered the book and plan to write a review later from windows internals and memory dump analysis perspective.

- Dmitry Vostokov @ DumpAnalysis.org -

Part 2 briefly discussed debuggers and their commands. Debugger commands can be grouped together into scripts that can be run against memory dump files and their resulted textual output can be redirected to log files.

Therefore we start with the first step in our analysis process diagram introduced in Part 1:

Information Collection: Scripts

Here a script is a sequence of instructions to extract formatted textual information from a memory dump file (or a live system). There are many advantages of scripts and logs. Many companies, for example, banks, avoid sending plain memory dump files because of security considerations in order to prevent exposure of company or private information. After the advent of 64-bit personal computing complete memory dump files became larger and larger and it is now common to get 32Gb memory dumps. Processing multiple long-running commands can easily be done from textual log files. Additionally, scripts can be used to process hundreds of memory dumps in one go to look for similarities and differences. Many tools can be used for such purpose including built-in debugger scripting capabilities, shell scripting and regular expressions for intelligent search.

- Dmitry Vostokov @ DumpAnalysis.org -

Before we explain stages of the analysis process shown in Part 1, let’s start with a brief overview of memory dumps, debuggers and logs. Recall that a memory dump is a snapshot of a process, system or physical memory state. This unifies post-mortem analysis and live debugging. Debuggers are tools that allow us to get and modify these memory snapshots. Other tools that allow us to get memory dump files are process dumpers like userdump.exe, Task Manager since Vista, WER, and system dumpers like LiveKd and Win32dd. We should not forget tools and methods that allow us to trigger Windows kernel ability to save consistent memory dump files: NMI button, keyboard method and various software bugcheck-triggers like Citrix SystemDump. Now coming back to debuggers. One of their essential features is to save a debugging session log, formatted textual output saved in a text file for further processing. One good example is !process 0 ff WinDbg command to output all processes and their thread stack traces (see Stack Trace Collection pattern for other variations). 

I’ve created a page to add all P-DMA parts as soon as I write them:

Pattern-Driven Memory Analysis

- Dmitry Vostokov @ DumpAnalysis.org -

Finally it has been published and available for orders from Amazon and other bookstores:

http://www.dumpanalysis.org/Debugged+Magazine

I had to increase the number of pages for the first issue from 16, planned originally, to 28 and this is reflected in the retail price of $10 (originally planned $8) but bookstores should sell it with a discount between 0% and 55%.

More information about the next issue should be ready by the end of the next week.

- Dmitry Vostokov @ DumpAnalysis.org - 

Previously announced Memory Analysis and Debugging Institute was registered in the Republic of Ireland (No. 382026) last week.

- Dmitry Vostokov @ DumpAnalysis.org - 

Yesterday I submitted the magazine to print and distribution world-wide. If everything is right it should be available by the end of this month. This first issue features 12 page WinDbg command supplement to pattern-driven memory dump analysis methodology, an overview of Win32dd complete memory dumper and PowerDbg enhancements to debug ASP.NET code. The magazine will only be available in print.

- Dmitry Vostokov @ DumpAnalysis.org -

Based on John Moore 8 science criteria we can consider Memory Dump Analysis (MDA) as a science:

1. MDA is based on data (memory dumps) collected in the field or re-pro / test environment.

2. Data (memory dumps) is collected to answer troubleshooting, debugging or forensics and intelligence questions. Observations in memory dumps are made to support or refute these questions.

3. Analysis of data (via memory dump analyzers, debuggers and log analyzers) is done objectively.

4. Troubleshooting, debugging or forensics hypotheses are developed and they are consistent with observations and compatible with general conceptual computer memory framework.

5. Troubleshooting, debugging or forensics hypotheses are tested and several comparable competing ones may be developed at any one time.

6. Generalizations are made that are valid universally within the domain of MDA.

7. The facts are confirmed independently.

8. Previously puzzling facts are explained.

It is also interesting to generalize the domain of MDA to empirical data collection via the so called universal memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org -

Last week I had an opportunity to present a pattern-driven memory dump analysis methodology at a global engineering conference. Now in a series of articles I’m going to clarify certain points and extend it to wider domain of memory analysis including computer memory forensics and intelligence.

Today I post the reworked picture of a waterfall-like analysis process:

Various phases and their relationship will be discussed in subsequent parts together with examples. 

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org accepts hardware such as laptops for reviewing in relation to their suitability for extreme debugging, computer forensics, crash dump analysis and memory visualization. If you work for a H/W company like HP, Apple, Dell, Acer, Sony or any other respectable manufacturer please don’t hesitate to forward this post to your management: it could be your company brand or laptop model that debugging and software technical support community chooses next time of upgrade or for T&D / R&D! H/W reviews will be posted on the main portal page which currently has an audience of more than a hundred thousand unique visitors per year from more than 20,000 network locations (*).

If your company is interested please don’t hesitate to use this contact form:

http://www.dumpanalysis.org/contact

(*) From Google Analytics report.

- Dmitry Vostokov @ DumpAnalysis.org -

I didn’t know that such challenge and contest exists until I came across this blog:

http://blog.flexilis.com/2008/12/the-2008-malware-challenge/

I’m always interested in malware and reverse engineering because sites about these topics usually contain hard-to-find Windows internals information. May be I try next year if such opportunity arises again.

- Dmitry Vostokov @ DumpAnalysis.org -

It had always been my dream since I left Moscow State University to be associated with a research institute. Until yesterday it became a reality with the announcement of

Memory Analysis & Debugging Institute (MA_&D^I).

www.madinstitute.com is under construction and currently points to memory dump analysis and debugging portal. Its structure, educational curriculum and research schools will be announced later. 

From: http://www.dumpanalysis.org/madinstitute-announcement

- Dmitry Vostokov @ DumpAnalysis.org -