Crash Dump Analysis Patterns (Part 3)

Another pattern I observe frequently is False Positive Dump. We get dumps pointing in a wrong direction or not useful for analysis and this usually happens when wrong tool was selected or right one was not properly configured for capturing crash dumps. Here is one example I investigated in detail.

The customer experienced frequent spooler crashes. The dump was sent for investigation to find an offending component: usually it is a printer driver. WinDbg revealed the following exception thread stack (parameters are not shown here for readability):

KERNEL32!RaiseException+0x56
KERNEL32!OutputDebugStringA+0x55
KERNEL32!OutputDebugStringW+0x39
HPZUI041!ConvertTicket+0x3c90
HPZUI041!DllGetClassObject+0x5d9b
HPZUI041!DllGetClassObject+0x11bb

The immediate response is to point to HPZUI041.DLL but if we look at parameters to KERNEL32!OutputDebugStringA we would see that the string passed to it is a valid NULL-terminated string:

0:010> da 000d0040
000d0040  ".Lower DWORD of elapsed time = 3"
000d0060  "750000."

If we disassemble OutputDebugStringA up to RaiseException call we would see:

0:010> u KERNEL32!OutputDebugStringA
KERNEL32!OutputDebugStringA+0x55
KERNEL32!OutputDebugStringA:
push    ebp
mov     ebp,esp
push    0FFFFFFFFh
push    offset KERNEL32!'string'+0x10
push    offset KERNEL32!_except_handler3
mov     eax,dword ptr fs:[00000000h]
push    eax
mov     dword ptr fs:[0],esp
push    ecx
push    ecx
sub     esp,228h
push    ebx
push    esi
push    edi
mov     dword ptr [ebp-18h],esp
and     dword ptr [ebp-4],0
mov     edx,dword ptr [ebp+8]
mov     edi,edx
or      ecx,0FFFFFFFFh
xor     eax,eax
repne scas byte ptr es:[edi]
not     ecx
mov     dword ptr [ebp-20h],ecx
mov     dword ptr [ebp-1Ch],edx
lea     eax,[ebp-20h]
push    eax
push    2
push    0
push    40010006h
call    KERNEL32!RaiseException

There is no jumps in the code prior to KERNEL32!RaiseException call and this means that raising exception was expected. Also MSDN documentation says:

“If the application has no debugger, the system debugger displays the string. If the application has no debugger and the system debugger is not active, OutputDebugString does nothing.”

So spoolsv.exe might have been monitored by a debugger which caught that exception and instead of dismissing it dumped the spooler process.

If we look at ‘analyze -v’ output we could see the following:

Comment: 'Userdump generated complete user-mode minidump
with Exception Monitor function on WS002E0O-01-MFP'
ERROR_CODE: (NTSTATUS) 0x40010006 -
Debugger printed exception on control C.

Now we see that debugger was User Mode Process Dumper you can download from Microsoft web site:

How to use the Userdump.exe tool to create a dump file 

If we download it, install it and write a small console program in Visual C++ to reproduce this crash:

#include "stdafx.h"
#include
int _tmain(int argc, _TCHAR* argv[])
{
    OutputDebugString(_T("Sample string"));
    return 0;
}

and if we compile it in Release mode and configure Process Dumper applet in Control Panel to include TestOutputDebugString.exe with the following properties:

and then run our program we would see Process Dumper catching KERNEL32!RaiseException and saving the dump.

Even if we select to ignore exceptions that occur inside kernel32.dll this tool still dumps our process. Now we can see that the customer most probably enabled ‘All Exceptions’ check box too. What the customer should have done is to use default rules like on the picture below:

Or select exception codes manually. In this case no dump is generated even if we manually select all of them. Just to check that the latter configuration still catches access violations we can add a line of code dereferencing NULL pointer and Process Dumper will catch it and save the dump.

Conclusion: the customer should have used NTSD as a default postmortem debugger from the start. Then if crash happened we would have seen the real offending component or could have applied other patterns and requested additional dumps.

- Dmitry Vostokov @ DumpAnalysis.org -

7 Responses to “Crash Dump Analysis Patterns (Part 3)”

  1. Management Bits and Tips » Blog Archive » Project Failure Analysis Patterns (Part 3) Says:

    […] mapping from False Positive Dump pattern brings us to False Project Failure pattern that usually happen when assessing the current […]

  2. !analyze -v : Crash Dump Analysis Patterns (Part 3) – False Positive Dump Says:

    […] http://www.dumpanalysis.org/blog/index.php/2006/11/01/crash-dump-analysis-patterns-part-3/ […]

  3. Aviv Greenberg Says:

    Thanks for posting this! it really saved me time because i just could not figure out from this dump i got what was wrong with the string that was passed - the memory was valid….

  4. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 5) Says:

    […] we introduce an icon for False Positive Dump […]

  5. Crash Dump Analysis » Blog Archive » Structural Memory Patterns (Part 1) Says:

    […] False Positive Dump […]

  6. Bob A Says:

    Thanks for the great article.
    I think I see a variant of this signature in another crash scenario:

    We’re seeing WER AppCrash reports, ending in
    Kernelbase!RaiseException
    OutputDebugString

    We’ve closely inspected our machines, and can’t see any debuggers present, nor does ‘!analyze -v’ mention one.
    The common theme in our issue is that we see this on Win Server 2008 R2 (64), where other OSes don’t fail (XP 32/64, Win7).
    We are a 32bit app.

    I suspect (and this is hunches) that in this OS, RaiseException is sometimes used to trigger a heap corruption check, we fail, and get terminated. Is this paranoia or FUD?

    Though (head/tail) pageheap and other lenses don’t seem to catch the corruption (it may be subtle?).

    Thanks again for a great blog, you’re an indispensable resource for Win developers!

  7. Dmitry Vostokov Says:

    I think you need to see what module calls OutputDebugString. You may also have other issues on other threads that !analyze -v is not able to figure out. You can ask for memory dump analysis audit:
    http://www.dumpanalysis.com/memory-dump-analysis-audit-service

Leave a Reply