Crash Dump Analysis Patterns (Part 41a)

Some memory dumps are generated on purpose to troubleshoot process and system hangs. They are usually called Manual Dumps, manual crash dumps or manual memory dumps. Kernel, complete and kernel mini dumps can be generated using the famous keyboard method described in the following Microsoft article which has been recently updated and contains the fix for USB keyboards:

http://support.microsoft.com/kb/244139

The crash dump will show E2 bugcheck:

MANUALLY_INITIATED_CRASH (e2)
The user manually initiated this crash dump.
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Various tools including Citrix SystemDump reuse E2 bug check code and its arguments.  There are many other 3rd-party tools used to bugcheck Windows OS such as BANG! from OSR or NotMyFault from Sysinternals. The old one is crash.exe that loads crashdrv.sys and uses the following bugcheck:

Unknown bugcheck code (69696969)
Unknown bugcheck description
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

In a memory dump you would see its characteristic stack trace pointing to crashdrv module: 

STACK_TEXT:
b5b3ebe0 f615888d nt!KeBugCheck+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
b5b3ebec f61584e3 crashdrv+0x88d
b5b3ec00 8041eec9 crashdrv+0x4e3
b5b3ec14 804b328a nt!IopfCallDriver+0x35
b5b3ec28 804b40de nt!IopSynchronousServiceTail+0x60
b5b3ed00 804abd0a nt!IopXxxControlFile+0x5d6
b5b3ed34 80468379 nt!NtDeviceIoControlFile+0x28
b5b3ed34 77f82ca0 nt!KiSystemService+0xc9
0006fed4 7c5794f4 ntdll!NtDeviceIoControlFile+0xb
0006ff38 01001a74 KERNEL32!DeviceIoControl+0xf8
0006ff70 01001981 crash+0x1a74
0006ff80 01001f93 crash+0x1981
0006ffc0 7c5989a5 crash+0x1f93
0006fff0 00000000 KERNEL32!BaseProcessStart+0x3d

Sometimes various hardware buttons are used to trigger NMI and generate a crash dump when keyboard is not available. The bugcheck will be:

NMI_HARDWARE_FAILURE (80)
This is typically due to a hardware malfunction. The hardware supplier should be called.
Arguments:
Arg1: 004f4454
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Critical process termination such as session 0 csrss.exe is used to force a memory dump:

CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been terminated.
Several processes and threads are necessary for the operation of the system; when they are terminated (for any reason), the system can no longer function.
Arguments:
Arg1: 00000003, Process
Arg2: 8a090d88, Terminating object
Arg3: 8a090eec, Process image file name
Arg4: 80967b74, Explanatory message (ascii)

- Dmitry Vostokov @ DumpAnalysis.org -

10 Responses to “Crash Dump Analysis Patterns (Part 41a)”

  1. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 73) Says:

    […] the case of the fully initialized system the manual dump might have been taken after reboot when the bugcheck already happened or any other reason stemming […]

  2. Crash Dump Analysis » Blog Archive » Manual and early crash dump, stack trace collection, main thread, blocked threads and pass through functions: pattern cooperation Says:

    […] - The Year of Debugging 2010 (0×7DA) - The Year of Dump Analysis The system was hanging and a manual kernel dump file was […]

  3. Dmitry Vostokov Says:

    If you see myfault on the stack then the dump was generated by NotMyFault sysinternals tool. I have started to see this from x64 dumps

    1: kd> k
    Child-SP RetAddr Call Site
    fffffa60`06bf7558 fffff800`0165712e nt!KeBugCheckEx
    fffffa60`06bf7560 fffff800`0165600b nt!KiBugCheckDispatch+0×6e
    fffffa60`06bf76a0 fffffa60`053da17a nt!KiPageFault+0×20b
    fffffa60`06bf7830 fffffa60`053da397 myfault+0×117a
    fffffa60`06bf7990 fffff800`018dd25a myfault+0×1397
    fffffa60`06bf79f0 fffff800`018f5f76 nt!IopXxxControlFile+0×5da
    fffffa60`06bf7b40 fffff800`01656e33 nt!NtDeviceIoControlFile+0×56
    fffffa60`06bf7bb0 00000000`77525aea nt!KiSystemServiceCopyEnd+0×13

  4. Dmitry Vostokov Says:

    Some virtualized environments may have their own methods to trigger a crash dump. For example, on XenServer we can see these bugchecks:

    x86:

    BugCheck D1, {f001, 2, 0, f001}

    DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
    An attempt was made to access a pageable (or completely invalid) address at an
    interrupt request level (IRQL) that is too high. This is usually
    caused by drivers using improper addresses.
    If kernel debugger is available get stack backtrace.
    Arguments:
    Arg1: 0000f001, memory referenced
    Arg2: 00000002, IRQL
    Arg3: 00000000, value 0 = read operation, 1 = write operation
    Arg4: 0000f001, address which referenced memory

    STACK_TEXT:
    805573c0 0000f001 nt!KiTrap0E+0x238
    80557430 ffdffc70 0xf001
    80557450 804dcbef 0xffdffc70
    80557454 00000000 nt!KiIdleLoop+0x10

    x64:

    BugCheck 1E, {ffffffffc0000005, f001, 8, f001}

    KMODE_EXCEPTION_NOT_HANDLED (1e)
    This is a very common bugcheck. Usually the exception address pinpoints
    the driver/function that caused the problem. Always note this address
    as well as the link date of the driver/image that contains this address.
    Arguments:
    Arg1: ffffffffc0000005, The exception code that was not handled
    Arg2: 000000000000f001, The address that the exception occurred at
    Arg3: 0000000000000008, Parameter 0 of the exception
    Arg4: 000000000000f001, Parameter 1 of the exception

    STACK_TEXT:
    fffff800`062b8358 fffff800`01896747 : nt!KeBugCheckEx
    fffff800`062b8360 fffff800`018b1ce9 : nt! ?? ::FNODOBFM::`string'+0x250e7
    fffff800`062b8960 fffff800`018b0ae5 : nt!KiExceptionDispatch+0xa9
    fffff800`062b8b40 00000000`0000f0: nt!KiPageFault+0x1e5
    fffff800`062b8cd8 fffffa60`02a7e685 : 0xf001
    fffff800`062b8ce0 fffff800`018b6583 : intelppm!C1Idle+0x9
    fffff800`062b8d10 fffff800`018b62a1 : nt!PoIdle+0x183
    fffff800`062b8d80 fffff800`01a88860 : nt!KiIdleLoop+0x21
    fffff800`062b8db0 00000000`fffff800 : nt!zzz_AsmCodeRange_End+0x4
    fffff800`062b20b0 00000000`00000000 : 0xfffff800

  5. Dmitry Vostokov Says:

    We can also see code f001 on the following thread stack as well:

    DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
    A device driver attempting to corrupt the system has been caught. This is
    because the driver was specified in the registry as being suspect (by the
    administrator) and the kernel has enabled substantial checking of this driver.
    If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
    be among the most commonly seen crashes.
    Arguments:
    Arg1: 0000000000000091, A driver switched stacks using a method that is not supported by
    the operating system. The only supported way to extend a kernel
    mode stack is by using KeExpandKernelStackAndCallout.
    Arg2: 0000000000000000
    Arg3: fffff8000185bc40
    Arg4: 0000000000000000

    0: kd> kL 100
    Child-SP RetAddr Call Site
    fffff880`0891f8c8 fffff800`01729c8a nt!KeBugCheckEx
    fffff880`0891f8d0 fffff800`01700573 nt! ?? ::FNODOBFM::`string'+0x4904
    fffff880`0891f910 fffff800`0170d8df nt!RtlDispatchException+0x33
    fffff880`0891fff0 fffff800`016d2c42 nt!KiDispatchException+0x16f
    fffff880`08920680 fffff800`016d17ba nt!KiExceptionDispatch+0xc2
    fffff880`08920860 fffff800`016ca533 nt!KiPageFault+0x23a
    fffff880`089209f8 fffff800`016a88c0 nt!memcpy+0x223
    fffff880`08920a00 fffff800`016a8638 nt!KiOpFetchBytes+0x30
    fffff880`08920a30 fffff800`0170dd6f nt!KiOpDecode+0x68
    fffff880`08920a80 fffff800`0170d896 nt!KiPreprocessFault+0x53
    fffff880`08920b10 fffff800`016d2c42 nt!KiDispatchException+0x126
    fffff880`089211a0 fffff800`016d17ba nt!KiExceptionDispatch+0xc2
    fffff880`08921380 00000000`0000f001 nt!KiPageFault+0x23a
    fffff880`08921518 fffff800`016d9b4b 0xf001
    fffff880`08921520 fffff800`016d94da nt!EnlightenedSwapContext_PatchXSave+0xa8
    fffff880`08921560 fffff800`016da752 nt!KiSwapContext+0x7a
    fffff880`089216a0 fffff800`016dc8af nt!KiCommitThreadWait+0x1d2
    fffff880`08921730 fffff800`0169c1de nt!KeWaitForSingleObject+0x19f
    fffff880`089217d0 fffff800`016db8bc nt!ExpWaitForResource+0xae
    fffff880`08921840 fffff880`042fcd91 nt!ExAcquireResourceExclusiveLite+0x14f
    [...]
    fffff880`08921a10 fffff800`019eff16 nt!IopXxxControlFile+0x607
    fffff880`08921b40 fffff800`016d2853 nt!NtDeviceIoControlFile+0x56
    fffff880`08921bb0 00000000`7755ff2a nt!KiSystemServiceCopyEnd+0x13
    00000000`0343f708 00000000`00000000 0x7755ff2a

  6. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 66) Says:

    […] Experts Magazine Online Today we introduce an icon for Manual Dump (kernel) […]

  7. Crash Dump Analysis » Blog Archive » Structural Memory Patterns (Part 1) Says:

    […] Manual Dump (kernel) […]

  8. Dmitry Vostokov Says:

    Memory dump from XenServer:
    http://support.citrix.com/article/CTX123177

  9. Dmitry Vostokov Says:

    Another example of hardware manual dump is BugCheck 8E, {c0000005, …

    BUCKET_ID: IP_MISALIGNED

    0: kd> .trap 0xffffffff99189ce4
    ErrCode = 00000002
    eax=00000115 ebx=8092596a ecx=00000000 edx=00cbff30 esi=00cbff38 edi=99189d64
    eip=8092596c esp=99189d58 ebp=99189d64 iopl=0 nv up ei ng nz na pe cy
    cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010287
    nt!NtUnmapViewOfSection+0x2:
    8092596c 8c28 mov word ptr [eax],gs ds:0023:00000115=????????

    0: kd> u nt!NtUnmapViewOfSection
    nt!NtUnmapViewOfSection:
    8092596a e9cd8c2877 jmp driverA+0x4263c (f7bae63c)
    8092596f 51 push ecx
    80925970 64a124010000 mov eax,dword ptr fs:[00000124h]
    80925976 8a80d7000000 mov al,byte ptr [eax+0D7h]
    8092597c 3c01 cmp al,1
    8092597e 56 push esi
    8092597f 8b750c mov esi,dword ptr [ebp+0Ch]
    80925982 8845fc mov byte ptr [ebp-4],al

    The server was reported hang and all CPUs were busy.

  10. Dmitry Vostokov Says:

    Memory Dump API capability in Windows 8.1
    http://crashdmp.wordpress.com/2014/08/01/windows-8-1-update-live-dump-capability/
    http://crashdmp.wordpress.com/2014/08/01/introducing-livedump-exe/

Leave a Reply

You must be logged in to post a comment.