Using scripts to process hundreds of user dumps

Forthcoming Webinar on Pattern-Driven Software Diagnostics

2012 - The Year of Software Trace Analysis

Accelerated Memory Dump Analysis Training

Sponsored link: Memory Dump Analysis Services

Debugging Experts Magazine Online

Debugging Today Daily Newspaper

Debugging TV

Suppose you have 100 - 200 user dumps from various user processes in the system and you want to quickly check their thread stacks, locks, etc. to see something suspicious related to your product or its environment your customers complaining about. It is much easier to collect such information into text files and browse them quickly than open every dump in WinDbg. I used shell script (VBScript) to automate loading dumps into WinDbg and used WinDbg scripts to run complex commands against loaded user dumps. For example, I used the following shell script:

'
' UDumps2Txt.vbs
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set Folder = fso.GetFolder(".")
Set Files = Folder.Files
Set WshShell = CreateObject("WScript.Shell")
For Each File In Files
  Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for Windows\WinDbg.exe -y ""srv*c:\mss*http://msdl.microsoft.com/download/symbols"" -z " + File.Name + " -c ""$$><c:\scripts\UDmp2Txt.txt;q"" -Q -QS -QY –QSY")
  Do While oExec.Status = 0
     WScript.Sleep 1000
  Loop
Next
'
' UDumps2Txt.vbs: End of File
'

and the following WinDbg script:

$$
$$ UDmp2Txt: Dump information from user dump into log
$$
.logopen /d
!analyze -v
!locks
~*kv
lmv
.logclose
$$
$$ UDmp2Txt: End of File
$$

The following command launches multiple Dmp2Txt conversions:

C:\UserDumps>cscript /nologo c:\scripts\UDumps2Txt.vbs

You can also use CDB from Debugging Tools for Windows (console debugger) instead of WinDbg. I just use WinDbg uniformly instead of using separately CDB for user process dumps and KD for kernel and complete memory dumps. 

Now when you have text files you can search for patterns using regular expressions. I will write more about applying them later. There is a very good book about them from practical point of view I read 6 years ago when I needed to understand them beyond wildcards and question marks. Since that time the book has undergone another two editions:

Mastering Regular Expressions, 3rd edition

Buy from Amazon

Or you can process text files further and feed them into your database - part of automated crash dump analysis system.

- Dmitry Vostokov -

Sponsored link: Professional Software Debugging Services

/* Malware and Software Defects -> Victimware.org */

Copyright © 2006 - 2012. This is a non-profit research and scientific project.

           

Citrix and Microsoft Customer Forum

Museum of Debugging and Memory Dumps

7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story

Crash and Hang Analysis Audit Service

CARE: Crash Analysis Report Environment

Crash Dump and Software Trace Analysis Training and Seminars

Access OpenTask Titles on Safari Books Online

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

International Memory Analysts and Debuggers Day:
07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA

Announcements

Coming Soon:

Resume and CV: As a Book

Fundamentals of Complete Crash and Hang Memory Dump Analysis

Management Bits: An Anthology from Reductionist Manager

Crash Dump Analysis: Practical Foundations (Windows Edition, Systematic Software Fault Analysis Series)

Crash Dump Analysis for System Administrators and Support Engineers

New Magazines:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers


New Books:

Advanced Windows Memory Dump Analysis with Data Structures: Training Course Transcript and WinDbg Practice Exercises with Notes

Accelerated .NET Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes

Accelerated Windows Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes

Introduction to Pattern-Driven Software Problem Solving

Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5

Windows Debugging Notebook: Essential User Space WinDbg Commands

Memory Dump Analysis Anthology, Volume 5

Memory Dump Analysis Anthology, Volume 4

Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3

Memory Dump Analysis Anthology, Volume 3

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

x64 Windows Debugging: Practical Foundations

Also available:

Windows Debugging: Practical Foundations

DLL List Landscape: The Art from Computer Memory Space

Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

WinDbg: A Reference Poster and Learning Cards

Memory Dump Analysis Anthology, Volume 2

Memory Dump Analysis Anthology, Volume 1

New Children's Book:

Baby Turing

3 Responses to “Using scripts to process hundreds of user dumps”

  1. gaurav Says:

    this script is not working for me
    C:\>UDumps2Txt.vbs
    C:\UDumps2Txt.vbs(10, 3) WshShell.Exec: The system cannot find the file specifie
    d.

  2. Dmitry Vostokov Says:

    Be sure that WshShell.Exec(…) is one line in the script file if you copied it from here. Check if it ws not split into several line after copy-pase and if WinDbg is in the same directory as specified.

  3. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 116) Says:

    […] user process memory dumps taken from several production servers, say 20 files, we can either employ scripts to process all of them or compare their file size and look for a bigger ones for a starter, for example, 85 or 110 Mb. […]

Leave a Reply