Crash Dump Analysis

Models for Memory and Trace Analysis Patterns (Part 1)

Sponsored link: Memory Dump Analysis Services

Due to many requests for memory dumps corresponding to crash dump analysis patterns I’ve started modeling software behaviour and defects. Every pattern will have an example application(s), service(s) or driver(s) or combination of them. Their execution results in memory layout that corresponds to memory or trace analysis patterns. Today we introduce an example model for Multiple Exceptions (user mode) pattern. The following source code models 3 threads each having an exception during their execution on Windows XP, Windows 7 and Windows Server 2008 R2:

// MultipleExceptions-UserMode // Copyright (c) 2010 Dmitry Vostokov // GNU GENERAL PUBLIC LICENSE // http://www.gnu.org/licenses/gpl-3.0.txt

#include <windows.h> #include <process.h>

void thread_one(void *) { *(int *)NULL = 0; }

void thread_two(void *) { *(int *)NULL = 0; }

int main(int argc, WCHAR* argv[]) { _beginthread(thread_two, 0, NULL); _beginthread(thread_one, 0, NULL);

DebugBreak();

return 0; }

In fact, thread_one and thread_two can be replaced with just one function because they are identical. Visual C++ compiler does that during code optimization. On Windows 7 and W2K8 R2 I created LocalDumps registry key to save full crash dumps. On Windows XP I set Dr. Watson as a postmortem debugger (via drwtsn32 -i command and configured it to save full user dumps via drwtsn32 command that brings Dr. Watson GUI). Vista had some peculiar behaviour so I postponed its discussion for another post. The application can be downloaded from here (zip file contains source code, x86 and x64 binaries together with corresponding PDB files):

Download MultipleExceptions-UserMode.zip

Now I provide modeling results for x64 W2K8 R2 running on 2 processor machine. Windows 7 and Windows XP results are very similar. If we run x64 executable it crashes and a dump file is saved (x86 crash dump is similar). Default analysis command gives these results:

0:000> !analyze -v

[...]

FAULTING_IP: MultipleExceptions_UserMode!thread_two+0 00000001`3f8b1000 c704250000000000000000 mov dword ptr [0],0

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 000007fefddc2442 (KERNELBASE!DebugBreak+0x0000000000000002) ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 1 Parameter[0]: 0000000000000000

[...]

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

[...]

PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT

[...]

STACK_TEXT: 00000001`3f8b1000 MultipleExceptions_UserMode!thread_two+0x0 00000001`3f8b10eb MultipleExceptions_UserMode!_callthreadstart+0x17 00000001`3f8b1195 MultipleExceptions_UserMode!_threadstart+0x95 00000000`778cf56d kernel32!BaseThreadInitThunk+0xd 00000000`77b03281 ntdll!RtlUserThreadStart+0x1d

[...]

We see debug break on the first thread:

0:000> kL Child-SP RetAddr Call Site 00000000`002eec78 000007fe`fdd913a6 ntdll!NtWaitForMultipleObjects+0xa 00000000`002eec80 00000000`778d3143 KERNELBASE!WaitForMultipleObjectsEx+0xe8 00000000`002eed80 00000000`77949025 kernel32!WaitForMultipleObjectsExImplementation+0xb3 00000000`002eee10 00000000`779491a7 kernel32!WerpReportFaultInternal+0x215 00000000`002eeeb0 00000000`779491ff kernel32!WerpReportFault+0x77 00000000`002eeee0 00000000`7794941c kernel32!BasepReportFault+0x1f 00000000`002eef10 00000000`77b6573c kernel32!UnhandledExceptionFilter+0x1fc 00000000`002eeff0 00000000`77ae5148 ntdll! ?? ::FNODOBFM::`string'+0x2365 00000000`002ef020 00000000`77b0554d ntdll!_C_specific_handler+0x8c 00000000`002ef090 00000000`77ae5d1c ntdll!RtlpExecuteHandlerForException+0xd 00000000`002ef0c0 00000000`77b1fe48 ntdll!RtlDispatchException+0x3cb 00000000`002ef7a0 000007fe`fddc2442 ntdll!KiUserExceptionDispatcher+0x2e 00000000`002efd58 00000001`3f8b103c KERNELBASE!DebugBreak+0×2 00000000`002efd60 00000001`3f8b13fb MultipleExceptions_UserMode!main+0×2c 00000000`002efd90 00000000`778cf56d MultipleExceptions_UserMode!__tmainCRTStartup+0×15b 00000000`002efdd0 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd 00000000`002efe00 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

2 other threads show exception processing too:

0:000> ~1s; kL ntdll!NtDelayExecution+0xa: 00000000`77b201fa c3 ret Child-SP RetAddr Call Site 00000000`0076ef78 000007fe`fdd91203 ntdll!NtDelayExecution+0xa 00000000`0076ef80 00000000`77949175 KERNELBASE!SleepEx+0xab 00000000`0076f020 00000000`779491ff kernel32!WerpReportFault+0×45 00000000`0076f050 00000000`7794941c kernel32!BasepReportFault+0×1f 00000000`0076f080 00000000`77b6573c kernel32!UnhandledExceptionFilter+0×1fc 00000000`0076f160 00000000`77ae5148 ntdll! ?? ::FNODOBFM::`string’+0×2365 00000000`0076f190 00000000`77b0554d ntdll!_C_specific_handler+0×8c 00000000`0076f200 00000000`77ae5d1c ntdll!RtlpExecuteHandlerForException+0xd 00000000`0076f230 00000000`77b1fe48 ntdll!RtlDispatchException+0×3cb 00000000`0076f910 00000001`3f8b1000 ntdll!KiUserExceptionDispatcher+0×2e 00000000`0076fec8 00000001`3f8b10eb MultipleExceptions_UserMode!thread_two 00000000`0076fed0 00000001`3f8b1195 MultipleExceptions_UserMode!_callthreadstart+0×17 00000000`0076ff00 00000000`778cf56d MultipleExceptions_UserMode!_threadstart+0×95 00000000`0076ff30 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd 00000000`0076ff60 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:001> ~2s; kL ntdll!NtDelayExecution+0xa: 00000000`77b201fa c3 ret Child-SP RetAddr Call Site 00000000`0086e968 000007fe`fdd91203 ntdll!NtDelayExecution+0xa 00000000`0086e970 00000000`77949175 KERNELBASE!SleepEx+0xab 00000000`0086ea10 00000000`779491ff kernel32!WerpReportFault+0×45 00000000`0086ea40 00000000`7794941c kernel32!BasepReportFault+0×1f 00000000`0086ea70 00000000`77b6573c kernel32!UnhandledExceptionFilter+0×1fc 00000000`0086eb50 00000000`77ae5148 ntdll! ?? ::FNODOBFM::`string’+0×2365 00000000`0086eb80 00000000`77b0554d ntdll!_C_specific_handler+0×8c 00000000`0086ebf0 00000000`77ae5d1c ntdll!RtlpExecuteHandlerForException+0xd 00000000`0086ec20 00000000`77b1fe48 ntdll!RtlDispatchException+0×3cb 00000000`0086f300 00000001`3f8b1000 ntdll!KiUserExceptionDispatcher+0×2e 00000000`0086f8b8 00000001`3f8b10eb MultipleExceptions_UserMode!thread_two 00000000`0086f8c0 00000001`3f8b1195 MultipleExceptions_UserMode!_callthreadstart+0×17 00000000`0086f8f0 00000000`778cf56d MultipleExceptions_UserMode!_threadstart+0×95 00000000`0086f920 00000000`77b03281 kernel32!BaseThreadInitThunk+0xd 00000000`0086f950 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We look at unhandled exception filter parameter to get exception pointers information:

0:002> kv Child-SP RetAddr : Args to Child : Call Site [...] 00000000`0086ea70 00000000`77b6573c : 00000000`0086ebb0 00000000`00000006 00000001`00000000 00000000`00000001 : kernel32!UnhandledExceptionFilter+0×1fc

0:002> .exptr 00000000`0086ebb0

----- Exception record at 00000000`0086f7f0: ExceptionAddress: 000000013f8b1000 (MultipleExceptions_UserMode!thread_two) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 0000000000000001 Parameter[1]: 0000000000000000 Attempt to write to address 0000000000000000

----- Context record at 00000000`0086f300: rax=00000000000ef0b0 rbx=00000000000ef0b0 rcx=0000000000000000 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000 rip=000000013f8b1000 rsp=000000000086f8b8 rbp=0000000000000000 r8=000007fffffda000 r9=0000000000000000 r10=0000000000000045 r11=000007fffffd9328 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz na pe nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 MultipleExceptions_UserMode!thread_two: 00000001`3f8b1000 c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=????????

What we now see that default analysis command showed the break instruction exception record and error code from the first thread but IP and stack trace from other threads having NULL pointer access violation exception.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Museum of Debugging and Memory Dumps

7/7/2010 - 8/8/2010 Annual Competition: Tell Your Windows Debugging Story

Crash and Hang Analysis Audit Service

CARE: Crash Analysis Report Environment

Crash Dump and Software Trace Analysis Training and Seminars

Access OpenTask Titles on Safari Books Online

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

International Memory Analysts and Debuggers Day:
07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA

Announcements

Coming Soon:

Management Bits: An Anthology from Reductionist Manager