Crash Dump Analysis

An exercise in de-analysis

This anti-pattern is called Wild Explanation. This is a free floating explanation based on loose associations. Its extreme version uses Gödel incompleteness theorems (undecidable crashes and hangs), quantum mechanics (in small time delta any bug can appear and disappear without being caught) or hydrodynamics (code fluidity, turbulence around processor cores) for abnormal software behaviour explanation (similar to Fashionable Nonsense in philosophy, humanities and social sciences). Its milder version is slightly modified original analysis monologue I found when searching Google for stack traces (I provide my comments below):

“Two hints make me assume this can be a physical memory problem:”

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except,
it must be protected by a Probe.  Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: e37cc20e, memory referenced.
Arg2: 00000001, value 0 = read operation, 1 = write operation.
Arg3: 8083fe2c, If non-zero, the instruction address which referenced the bad memory
     address.
Arg4: 00000000, (reserved)

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx” referenced memory at “0x%08lx”. The memory could not be “%s”.

I wouldn’t be so quick. Check Hardware Error pattern post and comments there. So let’s de-analyze the analysis.

“c0000005 is Access Denied where C is virtual memory, meaning usually disk cache”

c is just STATUS_SEVERITY_ERROR (0×3 or 0y11) 11000000000000000000000000000101 and doesn’t have any connection to cache or crash beyond a mere coincidence. See NTSTATUS bit descriptions: http://msdn.microsoft.com/en-us/library/cc231200(PROT.10).aspx

“It could be RAM and/or it could be savrt.sys if you’re using something like an antivirus.
Check to see if you need updates for them, Norton, Symantec, McCaffee.”

Of course, it is either RAM or antivirus, what else? What about a virus? All household names.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx” referenced memory at “0x%08lx”.  ”The memory could not be “%s”.” means there is something wrong with the debugger itself; those should be addresses, not percent signs and alphanumeric garbage.”

Now even a debugger is corrupt. RAM again? It is not a garbage, they’re format codes (like for printf C function) to print actual addresses.

“8083fe2c” this address called a pointer, that referenced “e37cc20e” again, that “e” has specific meaning, some virtual device or another, probably CD ROM, trying to transfer data to RAM.  This implies either SCSI or PCI buss or DMA buss timing problems.”

RAM again… Evidence mounts stronger and stronger. Its all about timing. A bus buzzes.

“Which is usually caused by a bad driver, not RAM, but sometimes replacing the RAM “masks” out the failure.”

You never know… A bit of doubt?

“This normally happens with 64-bit machines that take advantage of the 64-bit superfast address access and indexing, known as Vector Indirect Addressing, which is, apparently, too fast for even recent Operating System code.”

I always suspected these machines run x64 times faster… I can only imagine horrors when Vector Indirect Addressing hits an old Operating System code. However, the crash under discussion is x86…

“A write to the “e” device; so, is it some kind of CDR or DVDR or other storage device?”

I guess that was a rhetorical question… or let me finish this thread of thought to connect dots, e-devices, virtual memory, disk cache and CD ROM. What if a page file was configured on a DVD-R?

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):

• Are you interested in on-site training, prefer traveling or attending webinars?
• Are you interested in software trace analysis as well?
• What specific topics are you interested in?
• What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)

Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.

Looking forward to your responses. Any suggestions are welcome.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we follow the course of established troubleshooting or debugging methods without questioning them or not considering possible exceptions. For example, the usual advice to heap corruption signs in process memory dumps is to ask to enable full page heap. However page heap helps to catch buffer overwrites but not underwrites and heap can also be damaged by other means like double free, passing an invalid address or direct corruption of control structures via a dangling pointer. Therefore in some cases when we enable full page heap we get the same stack trace:

Normal:

0:030> kL 100
ChildEBP RetAddr 
175af77c 7d620ec5 ntdll_7d600000!RtlpCoalesceFreeBlocks+0×231
175af864 7c34218a ntdll_7d600000!RtlFreeHeap+0×38e
175af8ac 67d6153d msvcr71!free+0xc3
175af950 67781d69 Component!OnData+0×504
175af96c 7da4d03d Component!DispatchRequest+0×99
175afc28 7da4d177 rpcrt4!DispatchToStubInCNoAvrf+0×38
175afc7c 7da4d812 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
175afca0 7da59a1c rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
175afcfc 7da4da21 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×421
175afd20 7da3db14 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
175aff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
175aff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd
175affac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0×9d
175affb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0×1b
175affec 00000000 kernel32!BaseThreadStart+0×34

With full page heap enabled:

0:030> kL
ChildEBP RetAddr 
6092f568 7d6822fa ntdll!RtlpDphIsNormalHeapBlock+0×1f
6092f598 7d68256c ntdll!RtlpDphNormalHeapFree+0×21
6092f5f0 7d685443 ntdll!RtlpDebugPageHeapFree+0×146
6092f658 7d65714a ntdll!RtlDebugFreeHeap+0×2c
6092f730 7d62c5c0 ntdll!RtlFreeHeapSlowly+0×37
6092f814 7c34218a ntdll!RtlFreeHeap+0×11a
6092f85c 67d6153d msvcr71!free+0xc3
6092f900 67781d69 Component!OnData+0×504
6092f91c 7da4d03d Component!DispatchRequest+0×99
6092fc10 7da7ec5e rpcrt4!DispatchToStubInCNoAvrf+0×38
6092fc28 7da4d177 rpcrt4!DispatchToStubInCAvrf+0×14
6092fc7c 7da4d812 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
6092fca0 7da59a1c rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
6092fcfc 7da4da21 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×421
6092fd20 7da3db14 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
6092ff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
6092ff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd
6092ffac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0×9d
6092ffb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0×1b
6092ffec 00000000 kernel32!BaseThreadStart+0×34

We see that we get the same Component stack trace before calling free (shown in blue above) so we better search in a database of stack trace signatures before sending Habitual Reply to enable page heap. If there is a match in database then the course of future actions can be completely different: an immediate available hotfix or we find a similar problem that is already under investigation. The sooner we provide a relief to our customer the better.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes a memory dump is not useful because of missing memory space, hence the name of this anti-pattern: Missing Space. This usually happens for non-bugcheck type issues like CPU spikes, application or session hangs where user space analysis is needed. Here is an example of Spiking Thread pattern in a kernel memory dump where analysis cannot be done because threads were spiking mostly in user space not included in kernel memory dumps:

Loading Dump File [MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

1: kd> !running

System Processors 3 (affinity mask)
  Idle Processors 0

     Prcb      Current   Next   
  0  ffdff120  88d14b40            ................
  1  f7737120  891dddb0            ................

1: kd> !thread 88d14b40
THREAD 88d14b40  Cid 02b0.5298  Teb: 7ff7b000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    87e3ad50: (0006,0094) Flags: 00000900  Mdl: 00000000
    8a8485f8: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 e1001878
Owning Process            8a9831a8       Image:         ApplicationA.exe
Wait Start TickCount      1282379        Ticks: 0
Context Switch Count      2950            
UserTime                  00:00:15.109
KernelTime                00:00:00.125
Win32 Start Address 0×77c7b0f5
Start Address 0×77e617ec
Stack Init ae302000 Current ae301c38 Base ae302000 Limit ae2ff000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
0269fd6c 00000000 00000000 00000000 00000000 0×77e61684

1: kd> !thread 891dddb0
THREAD 891dddb0  Cid 2de4.3f58  Teb: 7ffd8000 Win32Thread: bc487c18 RUNNING on processor 1
Not impersonating
DeviceMap                 e6aba8b0
Owning Process            88b50d88       Image:         ApplicationB.exe
Wait Start TickCount      1282379        Ticks: 0
Context Switch Count      46241                 LargeStack
UserTime                  00:00:35.296
KernelTime                00:00:06.656
Win32 Start Address 0×0b80e53c
Start Address 0×77e617ec
Stack Init b234c000 Current b234bd00 Base b234c000 Limit b2346000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
b234bcc8 f65c0532 000000e2 00000000 00000000 nt!KeBugCheckEx+0×1b
b234bcf8 f65bfd2c 00866d40 bbf000c6 00000000 i8042prt!I8xProcessCrashDump+0×256
b234bd40 808395a5 8a107c28 8a866c88 0101000a i8042prt!I8042KeyboardInterruptService+0×225
b234bd40 432de484 8a107c28 8a866c88 0101000a nt!KiInterruptDispatch+0×49
WARNING: Frame IP not in any known module. Following frames may be wrong.
01bbf150 00000000 00000000 00000000 00000000 0×432de484

The solution here is to request either a complete memory dump or user dumps of identified processes. The former was requested already but the crash dump type in Control Panel was not checked before forcing the dump of the system via standard keyboard method. Therefore this is also an instance of Wrong Dump anti-pattern.

- Dmitry Vostokov @ DumpAnalysis.org -

This one is called No Question and is summarized by the following dialog that I observed many times:

As an engineer proceeds with analysis of a problem a phone rings and an irritated customer (internal or external) asks “What is the answer?”, an engineer glancing over the problem description replies, “What is the question?”.

This immediately came to my mind when I found this famous dialog:

“As the poet Gertrude Stein lay on her deathbed, her partner, Alice B. Toklas, leaned over and whispered, ‘What is the answer, Gertrude?’. Replied Stein, ‘What’s the question?’”

- Dmitry Vostokov @ DumpAnalysis.org -

Often engineers spend 10 minutes pursuing a certain investigation path and then prematurely closing it and switching to another. This is what I call Myopic Troubleshooting and Debugging.

This anti-pattern name was inspired by Daniel Dennett’s discussion of insufficiently patient scientists doing computer simulations:

“mistaking a failure of imagination for an insight into necessity” (Darwin’s Dangerous Idea, page 175).

Paraphrasing we can say that engineers think of impossibility where their imagination fails.

- Dmitry Vostokov @ DumpAnalysis.org -

Symbolless Analysis is another anti-pattern when an engineer either in a hurry or due to laziness doesn’t apply proper symbols and relies only on timestamps and module/offsets or trusts what WinDbg says and ignores symbol warnings. I usually apply symbols even in obvious cases and in hard ones strive to apply them until all possibilities are exhausted including search using PDBFinder.

Another weak variant is called Imageless Analysis when an engineer doesn’t specify proper Executable Image Search Path when it is necessary perhaps due to ignorance or just plain laziness again. Please see Minidump Analysis example for proper minidump analysis.

- Dmitry Vostokov @ DumpAnalysis.org -

This is sometime very funny one. It is called Fooled by Abbreviation. When someone is so presupposed or engaged in identifying Alien Components due to limited time and complexity of issues. For example, “Ctx” abbreviation in function names will most likely mean “Context” in general but can also be a function and data structure prefix used by a company with a similar sounding name. Opposite cases happen too when general is presupposed instead of particular, for example, ”Mms” prefix is read as “Memory Management Subsystem” but belongs to a multimedia system vendor. 

- Dmitry Vostokov @ DumpAnalysis.org -

Be language - excessive use of “is”. This anti-pattern was inspired by Alfred Korzybski notion of how “is” affects our understanding of the world. In the context of technical support the use of certain verbs sometimes leads to wrong troubleshooting and debugging paths. For example, the following phrase:

It is our pool tag. It is effected by driver A, driver B and driver C.  

Surely driver A, driver B and driver C were not developed by the same company that introduced the problem pool tag (smells Alien Component here). Unless supported by solid evidence the better phrase shall be:

It is our pool tag. It might have been effected by driver A, driver B or driver C.  

I’m not advocating to completely eradicate “be” verbs as was done in E-Prime language but to be conscious in their use. Thanks to Simple*ology in pointing me to the right direction.

- Dmitry Vostokov @ DumpAnalysis.org -

Need the crash dump. Period. This might be the first thought when an engineer gets a stack trace fragment without symbolic information. It is usually based on the following presupposition:

We need an actual dump file to suggest further troubleshooting steps.

This is not actually true unless it is the first time you have the problem and get stack trace for it. Consider the following fragment from bugcheck kernel dump when no symbols were applied because the customer didn’t have them:

b90529f8 8085eced nt!KeBugCheckEx+0x1b
b9052a70 8088c798 nt!MmAccessFault+0xb25
b9052a70 bfabd940 nt!_KiTrap0E+0xdc
WARNING: Stack unwind information not available. Following frames may be wrong.
b9052b14 bfabe452 MyDriver+0x27940

We can convert module+offset information into module!function+offset2 using MAP files or using DIA SDK (Debug Interface Access SDK) to query PDB files if we know module timestamp. This might be seen as a tedious exercise but we don’t need to do it if we keep raw stack trace signatures in some database when doing crash dump analysis. If we use our own symbol servers we might want to remove references to them and reload symbols. Then redo previous stack trace commands.

In my case it happened that I already analyzed similar previous bugcheck crash dumps months ago and saved stack trace prior to applying symbols. This helped me to point to solution without requesting the crash dump corresponding to that stack trace.

- Dmitry Vostokov @ DumpAnalysis.org -

Fooled by Description or the first incursion into Psychology of Memory Dump Analysis 

From my observation an engineer with software development background opens a crash dump after glancing at a problem description provided by a customer or even without reading it first. Only if the problem is not immediately obvious from the memory dump file the engineer will read the problem description thoroughly. On the contrary, an engineer with technical support or system administration background will thoroughly read the problem description first. In the latter case the description might influence the direction of analysis.

Here is an example. The description says - slow application start and you have a memory dump from a process. An engineer with technical support background will most likely look for hang patterns inside the dump. An engineer with experience writing native applications in C and C++ will open the dump and check an exception stored in it and if it is a breakpoint the suspicion might arise that the memory dump was taken manually because of the hanging process. Based on analysis the engineer might even correct the problem description or add questions that clarify the discrepancy between what is seen in the dump and what users perceive.

- Dmitry Vostokov @ DumpAnalysis.org -

A customer reports application.exe crashes and you ask for a dump file. You get a dump, open it and see the dump is not from your application.exe. You ask for print spooler crash dump and you get mplayer.exe crash dump. I originally thought to call it Wrong Dump pattern and place it into patterns category but after writing about Zippocricy I clearly see it as anti-pattern. It is not a rocket science to check process name in a dump file before sending it for analysis:

• Load the user process dump in WinDbg
• Type command .symfix; .reload; !analyze -v and wait

until WinDbg is not busy analyzing

• Find PROCESS_NAME: in the output. You get something like:

PROCESS_NAME: spoolsv.exe

You can also use dumpchk.exe from Debugging Tools for Windows.

I’m also writing a new version of Citrix DumpCheck Explorer extension that will include process name in its output.  

Another example is when you ask for a complete memory dump but you get a kernel dump or you get various mini-dumps. Fortunately DumpCheck extension can  warn users before they submit a dump.

- Dmitry Vostokov -

I have heard engineers saying, “I didn’t know about this debugging command, let’s use it!” after training session or reading other people’s analysis of crash dumps. A year later I hear the same phrase from them about another debugging command. In the mean time they continue to use the same set of commands they know about until they hear the old new one.

This is a manifestation of Word of Mouth anti-pattern.

General solution: Know your tools. Study them proactively. RTFM.

Example solution: periodically read and re-read WinDbg help.

More refined solution: debugger.chm on Windows Mobile PC.

- Dmitry Vostokov -

Let’s define Zippocricy - common sin in software support environments worldwide: someone gets something from a customer in archived form and without checking the contents forwards it further to another person in support chain. By the time the evidence gets unzipped somewhere, checked and found corrupt or irrelevant the customer suffers not hours but days.

Happens not only with crash dumps but with any type of problem evidence. 

- Dmitry Vostokov -

In any domain of activity where patterns exist we can find anti-patterns too. They are bad solutions for recurrent problems in specific contexts. One of them I would like to introduce briefly is Alien Component. In essence, when every technique fails or you run out of WinDbg commands look at some innocent component you have never seen before or don’t have symbols for: be it some driver or hook. Of course, this component cannot be the component developed by the company you are working for.

- Dmitry Vostokov -