Crash Dump Analysis

CARE: Crash Analysis Report Environment

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

This is a follow up to my previous Q&A about crash dumps and security issues like exposing confidential information stored in memory: Crash Dumps and Security. It seems a solution exists which allows to do some sort of crash dump analysis or at least identify problem components without sending complete or kernel memory dumps.

This solution takes advantage of WinDbg ability to execute scripts of arbitrary complexity. Couple of months ago I wrote about scripts and they really help me in pulling out various information from complete memory dumps:

WinDbg scripts
Yet another Windbg script
Critical sections

Now I created the bigger script that combines together all frequent commands used for identification of potential problems in memory dumps:

• !analyze -v
• !vm 4
• lmv
• !locks
• !poolused 3
• !poolused 4
• !exqueue f
• !irpfind
• !stacks
• List of all processes’ thread stacks, loaded modues and critical sections (for complete memory dump)

Other commands can be added if necessary.

How does all this work? A customer has to install Debugging Tools for Windows from Microsoft. This can be done on any workstation and not necessarily in a production environment. Then the customer has to run WinDbg.exe with some parameters including path(s) to symbols (-y), a path to memory dump (-z) and a path to script (-c):

C:\Program Files\Debugging Tools for Windows>WinDbg.exe -y "srv*c:\mss*http://msdl.microsoft.com/download/symbols" -z MEMORY.DMP -c "$$><c:\WinDbgScripts\Dmp2Txt.txt;q" -Q -QS -QY –QSY

Once WinDbg.exe finishes (it can run for couple of hours if you have many processes in your complete memory dump) you can copy the .log file created in “C:\Program Files\Debugging Tools for Windows” folder, archive it and send it to support for analysis. Kernel and process data and cached files are not exposed in the log! And because this is a text file the customer can inspect it before sending.

Here are the contents of Dmp2Txt.txt file:

$$
$$ Dmp2Txt: Dump all necessary information from complete full memory dump into log
$$
.logopen /d
!analyze -v
!vm 4
lmv
!locks
!poolused 3
!poolused 4
!exqueue f
!irpfind
!stacks
r $t0 = nt!PsActiveProcessHead
.for (r $t1 = poi(@$t0); (@$t1 != 0) & (@$t1 != @$t0); r $t1 = poi(@$t1))
{
    r? $t2 = #CONTAINING_RECORD(@$t1, nt!_EPROCESS, ActiveProcessLinks);
    .process @$t2
    .reload
    !process @$t2
    !ntsdexts.locks
    lmv
}
.logclose
$$
$$ Dmp2Txt: End of File
$$

For kernel dumps the script is simpler: 

$$
$$ KeDmp2Txt: Dump all necessary information from kernel dump into log
$$
.logopen /d
!analyze -v
!vm 4
lmv
!locks
!poolused 3
!poolused 4
!exqueue f
!irpfind
!stacks
!process 0 7
.logclose
$$
$$ KeDmp2Txt: End of File
$$


Note: if the dump is LiveKd.exe generated then due to inconsistency scripts may run forever 

- Dmitry Vostokov -

           

Announcements

Coming Soon:

Debugging Notebook: Essential Concepts, WinDbg Commands and Tools

Crash Dump Analysis for System Administrators and Support Engineers

New Magazines:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers

New Books:

Memory Dump Analysis Anthology, Volume 3

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

x64 Windows Debugging: Practical Foundations

Also available:

Windows Debugging: Practical Foundations

DLL List Landscape: The Art from Computer Memory Space

Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

WinDbg: A Reference Poster and Learning Cards

Memory Dump Analysis Anthology, Volume 2

Memory Dump Analysis Anthology, Volume 1

New Children's Book:

Baby Turing

This entry was posted on Saturday, December 9th, 2006 at 6:58 am and is filed under Crash Dump Analysis, Security, Tools, WinDbg Scripts. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.