Another pattern I observe frequently is False Positive Dump. We get dumps pointing in a wrong direction or not useful for analysis and this usually happens when wrong tool was selected or right one was not properly configured for capturing crash dumps. Here is one example I investigated in detail.
The customer experienced frequent spooler crashes. The dump was sent for investigation to find an offending component: usually it is a printer driver. WinDbg revealed the following exception thread stack (parameters are not shown here for readability):
KERNEL32!RaiseException+0x56
KERNEL32!OutputDebugStringA+0x55
KERNEL32!OutputDebugStringW+0x39
HPZUI041!ConvertTicket+0x3c90
HPZUI041!DllGetClassObject+0x5d9b
HPZUI041!DllGetClassObject+0x11bb
The immediate response is to point to HPZUI041.DLL but if we look at parameters to KERNEL32!OutputDebugStringA we would see that the string passed to it is a valid NULL-terminated string:
0:010> da 000d0040
000d0040 ".Lower DWORD of elapsed time = 3"
000d0060 "750000."
If we disassemble OutputDebugStringA up to RaiseException call we would see:
0:010> u KERNEL32!OutputDebugStringA
KERNEL32!OutputDebugStringA+0x55
KERNEL32!OutputDebugStringA:
push ebp
mov ebp,esp
push 0FFFFFFFFh
push offset KERNEL32!'string'+0x10
push offset KERNEL32!_except_handler3
mov eax,dword ptr fs:[00000000h]
push eax
mov dword ptr fs:[0],esp
push ecx
push ecx
sub esp,228h
push ebx
push esi
push edi
mov dword ptr [ebp-18h],esp
and dword ptr [ebp-4],0
mov edx,dword ptr [ebp+8]
mov edi,edx
or ecx,0FFFFFFFFh
xor eax,eax
repne scas byte ptr es:[edi]
not ecx
mov dword ptr [ebp-20h],ecx
mov dword ptr [ebp-1Ch],edx
lea eax,[ebp-20h]
push eax
push 2
push 0
push 40010006h
call KERNEL32!RaiseException
There is no jumps in the code prior to KERNEL32!RaiseException call and this means that raising exception was expected. Also MSDN documentation says:
“If the application has no debugger, the system debugger displays the string. If the application has no debugger and the system debugger is not active, OutputDebugString does nothing.”
So spoolsv.exe might have been monitored by a debugger which caught that exception and instead of dismissing it dumped the spooler process.
If we look at ‘analyze -v’ output we could see the following:
Comment: 'Userdump generated complete user-mode minidump
with Exception Monitor function on WS002E0O-01-MFP'ERROR_CODE: (NTSTATUS) 0x40010006 -
Debugger printed exception on control C.
Now we see that debugger was User Mode Process Dumper you can download from Microsoft web site:
How to use the Userdump.exe tool to create a dump file
If we download it, install it and write a small console program in Visual C++ to reproduce this crash:
#include "stdafx.h"
#include
int _tmain(int argc, _TCHAR* argv[])
{
OutputDebugString(_T("Sample string"));
return 0;
}
and if we compile it in Release mode and configure Process Dumper applet in Control Panel to include TestOutputDebugString.exe with the following properties:
and then run our program we would see Process Dumper catching KERNEL32!RaiseException and saving the dump.
Even if we select to ignore exceptions that occur inside kernel32.dll this tool still dumps our process. Now we can see that the customer most probably enabled ‘All Exceptions’ check box too. What the customer should have done is to use default rules like on the picture below:
Or select exception codes manually. In this case no dump is generated even if we manually select all of them. Just to check that the latter configuration still catches access violations we can add a line of code dereferencing NULL pointer and Process Dumper will catch it and save the dump.
Conclusion: the customer should have used NTSD as a default postmortem debugger from the start. Then if crash happened we would have seen the real offending component or could have applied other patterns and requested additional dumps.
- Dmitry Vostokov @ DumpAnalysis.org -
