Crash Dump Analysis

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

While I was listening to Klaus Schulze In Blue album a colleague sent me the link to a tool that reconstructs blue screens from minidumps (small memory dumps):

BlueScreenView (written by Nir Sofer)

I immediately downloaded it at it works even with kernel dumps but without pointing to a module that triggered the bugcheck (it shows modules for minidumps):

It ignores memory dumps and minidumps from x64 Windows so the next version I hope should do it

PS. Long time ago I was thinking about writing a kernel driver that saves BSOD screen and embeds it in a memory dump.

- Dmitry Vostokov @ DumpAnalysis.org -

I was in McDonald’s today with my daughter. This time they popularize Einstein, giving his stature in happy meal packs, telling that the size of his brain was the same as mine. My brain continued to work after meal and I finally understood that the right memory dump is what really matters. Your computer may have 1Tb of memory but if you didn’t get the right dump at the right moment you wasted your time.

- Dmitry Vostokov @ DumpAnalysis.org - 

DATA (Dump Artifact + Trace Artifact) - > DATA (Dump Analysis + Trace Analysis) examples:

1.  Efficient

- My 64Gb server bluescreens. I set a complete memory dump option in Control Panel.

- A user cannot connect. I started tracing yesterday. Stopped today.

- I analyze all these artifacts every day.

2. Effective

- My 64Gb server bluescreens. I set a kernel memory dump option in Control Panel.

- A user cannot connect. I started tracing, tried to connect, stopped tracing.

- I analyze all these artifacts every day and write articles to reduce DATA load.

- Dmitry Vostokov @ DumpAnalysis.org -

Inspired by Penrose tensor notation encountered in The Road to Reality book and Feynman diagrams I’d like to introduce Visual Dump Objects (VDO) graphical notation to depict and communicate memory dump analysis patterns, their combinations and analysis results. Let’s look at some basic visual objects (shown in handwriting).

1. A thread:

   or   

2. A function: 

3. A module:

4. A thread running through functions, modules or both (stack trace). Optional arrowhead can indicate stack trace direction:

  or    or  

Threads running through modules depict collapsed stack traces.

5. A blocked thread:

An example of 3 threads blocked by another thread (an arrowhead can disambiguate the direction of the waiting chain):

6. A spiking thread (colors are encouraged in VDO notation):

   or   

7. Space boundary between user land and kernel land:

Here is an example of the thread spiking in kernel space:

or with modules from stack trace:

More notation to come very soon.

 - Dmitry Vostokov @ DumpAnalysis.org -

Universal memory dumps come from astronomical observations:

Dumps “are meaningless without” symbols “to interpret them.”

Raymond Arthur Lyttleton, Quoted in Astroparticle Physics, page 50

- Dmitry Vostokov @ DumpAnalysis.org -

OpenTask, iterative and incremental publisher, plans to release a guide for general computer users explaining crashes and hangs on personal computers ranging from mobile phones, netbooks and laptops to desktops and servers:

My Computer Crashes and Freezes: A Non-technical Guide to Software and Hardware Errors (ISBN: 978-19067171704)

- Dmitry Vostokov @ DumpAnalysis.org -

Reading stack traces like English verse (remeber to read from bottom to top):

0:01> ~8kL
ChildEBP RetAddr 
009ef258 7c827d0b ntdll!KiFastSystemCallRet
009ef25c 7c83d236 ntdll!NtWaitForSingleObject+0xc
009ef298 7c83d281 ntdll!RtlpWaitOnCriticalSection+0x1a3
009ef2b8 7c82dabf ntdll!RtlEnterCriticalSection+0xa8
009ef358 7c82dab1 ntdll!LdrpGetProcedureAddress+0x128
009ef374 77e764ea ntdll!LdrGetProcedureAddress+0x18
009ef5d8 7c34c456 kernel32!UnhandledExceptionFilter+0x46f
009ef5f4 7c34957c msvcr71!_XcptFilter+0x15f
009ef600 7c34246e msvcr71!_endthreadex+0xb7
009ef628 7c828752 msvcr71!_except_handler3+0x61
009ef64c 7c828723 ntdll!ExecuteHandler2+0x26
009ef6f4 7c82855e ntdll!ExecuteHandler+0x24
009ef6f4 7c82be3e ntdll!KiUserExceptionDispatcher+0xe
009efa00 7c82a319 ntdll!RtlpFindEntry+0x68
009efc2c 7c3416b3 ntdll!RtlAllocateHeap+0x606
009efc6c 7c3416db msvcr71!_heap_alloc+0xe0
009efc74 7c360947 msvcr71!_nh_malloc+0x10
009efc80 0285f893 msvcr71!operator new+0xb
009efca8 02852e38 SQLModule!ODBCDelete+0xf3
009efd54 0269acff Store!ProcessDeletes+0x3d
009eff38 0269badb Store!UpdateStore+0xe
009eff58 00323499 Common!WorkItem+0x15c
009eff84 7c349565 Common!WorkItemThread+0x339
009effb8 77e64829 msvcr71!_endthreadex+0xa0
009effec 00000000 kernel32!BaseThreadStart+0x34

The new thread started
To work through items
It got an item
Handled it to store
To run delete requests
Through Oh-Dee-Bee-See
It tried to alloc
But crashed in malloc
While browsing the heap
Exception was dispatched
And handler called at once
But couldn’t find a filter
And called default one
That filter needed help
And looked for its address
But halted in suspense
While entering crit sec.

- Dmitry Vostokov @ DumpAnalysis.org -

I very proud to announce that after 3 weeks of final work the book has been released in both paperback and PDF format. In a week or so it should also appear on Amazon and other booksellers around the world. The book information and how to buy it can be found on the portal:

Windows Debugging: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org -

Crash “must be distinguished from” hang “with which it is often confounded.”

Sydney Smith, A Memoir of the Reverend Sydney Smith by his daughter, Lady Holland, with a Selection from his Letters

- Dmitry Vostokov @ DumpAnalysis.org -

As one of the new initiatives for the Year of Debugging  DumpAnalysis Portal will publish bimonthly full color 16 page publication called:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The only serial publication dedicated entirely to Windows® debugging

The first issue is planned for March, 2009 and will have ISBN-13: 978-1-906717-38-4. If it goes well I’m planning to have ISSN number assigned to it too. More details will be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org - 

I’m pleased to announce that OpenTask has submitted the book Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov for printing and here is the link to TOC:

Table of Contents

- Dmitry Vostokov @ DumpAnalysis.org - 

DLL is also a recursive acronym for DLL List Landscape. OpenTask is going to publish soon the new full color book:

Title: DLL List Landscape: The Art from Computer Memory Space
ISBN-13: 978-1-906717-36-0

More details will be announced tomorrow.  

- Dmitry Vostokov @ DumpAnalysis.org -

This “Hamlet’s Question” of software technical support is often asked and unfortunately sometimes not even asked at all when troubleshooting and debugging complex enterprise environments. For applications the question of saving crash dumps is trivial. If a process is not in memory and not visible in Task Manager we won’t be able to dump it manually. With OS always running even when hanging the question often degenerates to “Let’s bugcheck and send the crash dump to dump file divers”. After that decision huge amounts of energy are spent in collecting, sending and storing gigabytes of data with always very little or no return. Therefore here is the preliminary list of symptoms where manual system dumps are appropriate and when they are not:

When a manual system dump is appropriate

• - The system hangs visually (no GUI activity possible)

• - No connections or logins are possible

• - Abnormal system metrics (like pool, thread or process number growth)

• - Insufficient system or session memory

When a manual process user dump is more appropriate than a complete memory dump

• - Process hangs visually (other applications work as normal)

• - Error message box appears

• - Abnormal process metrics (like process memory growth or handle leaks)

When manual kernel and complete memory dumps are almost useless (I say almost because in rare circumstances they can aid in problem resolution so it is better not to collect them until explicitly asked from skilled memory dump file diver)

• - Application failures resulted in their disappearance from the list of running processes

• - Functional bugs (dynamic activity that requires historical tracing of events)

Note: 3rd-party kernel mode software developers should not face this question during the development of their drivers and delegate the responsibility for difficult bugcheck or panic decisions to an operating system. Surely Windows core developers face this question too.

Next we discuss another related question about choosing between kernel and complete memory dump options in Control Panel. 

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced book Crash Dump Analysis for System Administrators and Support Engineers (Windows Edition) has got its draft cover featuring WinDbg  output from a kernel memory dump forced by Citrix SystemDump tool.

Front:

Back:

- Dmitry Vostokov @ DumpAnalysis.org -

Paperback edition of Memory Dump Analysis Anthology, Volume 2 is finally available on Amazon and Barnes & Noble. Search Inside is also available on Amazon. In addition, I updated the list of recommended books:

Listmania! Crash Dump Analysis and Debugging

Hardcover edition will be available on Amazon and B&N in 2-3 weeks.

- Dmitry Vostokov @ DumpAnalysis.org -

“Everything is memory dump.”

I’m very excited to announce that Volume 2 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 2

In one or two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available by the end of October.

I’m often asked when Volume 3 is available and I currently plan to release it in October - November, 2009. In the mean time I’m planning to concentrate on other publishing projects. 

- Dmitry Vostokov @ DumpAnalysis.org -

The book is nearly finished and here is the final TOC:

Memory Dump Analysis Anthology, Volume 2: Table of Contents

- Dmitry Vostokov @ DumpAnalysis.org -