Software Diagnostics Library

Jamie Fenton has finally made available for beta testing fantastic memory visualization system HayWire & Barn Burner I use often for mining curious pictures from virtual memory space, like “cosmic rays“. Besides providing artistic pleasure, real-time natural memory visualization can have other applications including visual debugging. Looking forward to seeing more wonders in this area of research!

- Dmitry Vostokov @ DumpAnalysis.org -

Found today on Amazon that one seller sells cheap chopped copies of Memory Dump Analysis Anthology:

“The LOWEST PRICE because the spine binding & glue has been CHOPPED OFF; the binding is MISSING; this makes the loose pages suitable for photocopying or for hole punching to place into a 3-ring notebook. The pages have no marks or highlights. I also have Volume 2 with a cut spine for cheap.”

The seller has been contacted to stop this advertisement because the page number 2 in both volumes says:

You must not circulate this book in any other binding or cover and you must impose the same condition on any acquirer.

There is also the standard clause about reproduction and storage.

- Dmitry Vostokov @ DumpAnalysis.org -

Here is a bit of history of Memory Dump Analysis Anthology. Back in 2008, in January-February I was in search of the title name and was focused on variations of Crash Dump Analysis until one day I stumbled across this book in the local book shop (UK version was available earlier in February before US and you can still buy it on Amazon UK):

Memory: An Anthology

I immediately understood how I needed to name the collection of blog posts. Thus Memory Dump Analysis Anthology, Volume 1 was born.

- Dmitry Vostokov @ DumpAnalysis.org -

I resumed this week my reading notebook on Software Generalist blog with a top priority book to read every working day: Windows Internals, 5th edition. In reading notes I put what I find interesting for me (at this time) or related to Windows memory dump analysis or debugging and troubleshooting in general. For the latter case, sometimes I put additional references or even WinDbg examples from user, kernel and complete memory dumps in full color. Hope you find these notes useful too.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes I see engineers use memory “space” and protection “mode” terms interchangeably although, in my opinion, there is a big difference between them. They are independent from each other (orthogonal). For example, a thread may access a user space address but be running in a kernel protected mode. Even it is possible to call a user space function from a kernel space thread (that function shouldn’t try to call Windows API). The following combinations are possible:

user space / user mode
kernel space / kernel mode
user space / kernel mode

Most of the time space and mode coincide (blue - user, red - kernel):

ChildEBP RetAddr 
b6046c50 80833e95 nt!KiSwapContext+0×26
b6046c7c 8082b72b nt!KiSwapThread+0×2e5
b6046cc4 808ef652 nt!KeRemoveQueue+0×417
b6046d48 8088b19c nt!NtRemoveIoCompletion+0xdc
b6046d48 7c94860c nt!KiFastCallEntry+0xfc
00bfff70 7c9477f9 ntdll!KiFastSystemCallRet
00bfff74 7c959f68 ntdll!NtRemoveIoCompletion+0xc
00bfffb8 7c82482f ntdll!RtlpWorkerThread+0×3d
00bfffec 00000000 kernel32!BaseThreadStart+0×34

I personally prefer to talk about ”spaces” when I analyze complete memory dumps and almost never talk about “modes”. 

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we get crashes with the following stack trace pattern:

0:000> kv 100
ChildEBP RetAddr  Args to Child             
0012cecc 7c91df4a 7c8094fe 00000002 0012cef8 ntdll!KiFastSystemCallRet
0012ced0 7c8094fe 00000002 0012cef8 00000001 ntdll!ZwWaitForMultipleObjects+0xc
0012cf6c 7c80a085 00000002 0012d09c 00000000 kernel32!WaitForMultipleObjectsEx+0x12c
0012cf88 6990763c 00000002 0012d09c 00000000 kernel32!WaitForMultipleObjects+0x18
0012d91c 699082b1 0012f090 00000001 00198312 faultrep!StartDWException+0×5df
0012e990 7c8635d1 0012f090 00000001 00000000 faultrep!ReportFault+0×533
0012f030 78138a09 0012f090 00000022 c000000d kernel32!UnhandledExceptionFilter+0×587
0012f368 781443d1 00000000 00000000 00000000 msvcr80!_invoke_watson+0xc4
0012f38c 0040b02f 0012f538 00000104 004f80a0 msvcr80!strcat_s+0×29
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f39c 0012f538 00000104 00430848 0012f538 Application+0xb020

0:000> .exptr 0012f090

----- Exception record at 0012f040:
ExceptionAddress: 781443d1 (msvcr80!strcat_s+0x00000029)
   ExceptionCode: c000000d
  ExceptionFlags: 00000000
NumberParameters: 0

----- Context record at 0012f098:
eax=f2c4dacf ebx=00000000 ecx=00000002 edx=7c91e514 esi=00000022 edi=00000000
eip=781443d1 esp=0012f36c ebp=004f7658 iopl=0 nv up ei ng nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000286
msvcr80!strcat_s+0x29:
781443d1 83c414          add     esp,14h

0:000> !error c000000d
Error code: (NTSTATUS) 0xc000000d (3221225485) - An invalid parameter was passed to a service or function.

We clearly see that the crash involved strcat_s but we don’t see any invalid instruction or access violation call. The ADD instruction is perfectly valid above. However, if we disassemble EIP backwards we would see the call to _invalid_parameter function and it looks like it has a name association with Dr. Watson:

0:000> ub 781443d1
msvcr80!strcat_s+0×1c:
781443c4 5e              pop     esi
781443c5 8930            mov     dword ptr [eax],esi
781443c7 53              push    ebx
781443c8 53              push    ebx
781443c9 53              push    ebx
781443ca 53              push    ebx
781443cb 53              push    ebx
781443cc e89f46ffff      call    msvcr80!_invalid_parameter (78138a70)

0:000> uf 78138a70
Flow analysis was incomplete, some code may be missing
msvcr80!_invoke_watson:
78138945 55              push    ebp
78138946 8dac2458fdffff  lea     ebp,[esp-2A8h]
7813894d 81ec28030000    sub     esp,328h

[...]

msvcr80!_invalid_parameter:
78138a70 55              push    ebp
78138a71 8bec            mov     ebp,esp
78138a73 ff3528401c78    push    dword ptr [msvcr80!__pInvalidArgHandler (781c4028)]
78138a79 e85ba1ffff      call    msvcr80!_decode_pointer (78132bd9)
78138a7e 85c0            test    eax,eax
78138a80 59              pop     ecx
78138a81 7403            je      msvcr80!_invalid_parameter+0×16 (78138a86)

msvcr80!_invalid_parameter+0x13:
78138a83 5d              pop     ebp
78138a84 ffe0            jmp     eax

msvcr80!_invalid_parameter+0x16:
78138a86 6a02            push    2
78138a88 e806330000      call    msvcr80!_crt_debugger_hook (7813bd93)
78138a8d 59              pop     ecx
78138a8e 5d              pop     ebp
78138a8f e9b1feffff      jmp     msvcr80!_invoke_watson (78138945)

According to MSDN documentation, _s secure functions by default use a postmortem debugger mechanism:

Secure-enhanced CRT parameter validation

So we have something that is similar to Self-Dump pattern here. The same parameter checking is seen in the case of C++ STL exceptions. In case of custom unhandled exception filters not resorting to WER faulrep.dll other stacks can show process termination, for example, with wcscpy_s:

0:000> kL 100
ChildEBP RetAddr 
0111cb64 7c947c39 ntdll!KiFastSystemCallRet
0111cb68 7c80202b ntdll!ZwTerminateProcess+0xc
0111cb78 78138a2b kernel32!TerminateProcess+0×20
0111ceb4 78144ba1 MSVCR80!_invoke_watson+0xe6
0111ced8 67dbb47d MSVCR80!wcscpy_s+0×29
0111cf00 67dbc93b Application!CopyName+0×5d
[…]


Here the specified size of the destination buffer was smaller than the size of source NULL-terminated strings, which was a good thing anyway: old strcpy or strcat function would definitely caused buffer overflow effects. Now we have a nice side effect too, the dump is saved, ready for a postmortem analysis and subsequent code improvement.

- Dmitry Vostokov @ DumpAnalysis.org -

Just finished listening twice to Rick Wakeman performance at Lincoln Cathedral. To paraphrase his words, when I do memory dump analysis I experience the feeling of power surge through the fingers. Highly recommended to get the feeling of being the debugging god back during bitter moments of software support:

At Lincoln Cathedral

Here is my version of track titles inspired by listening (with my comments in italics):

1. Process Mortality
2. Dance of the MIPS
3. Gifts from Help
4. THE ALL MIGHTY DEBUGGER
5. Problem solved
6. The DA TA Variations (remember DATA: Dump Analysis Trace Analysis)

- Dmitry Vostokov @ DumpAnalysis.org -

One 32-bit WOW64 process was crashing when accessing a direct NULL data pointer with the following stack trace:

0:000> r
rax=00000000750e9b40 rbx=000000000017f29c rcx=0000000000041710
rdx=0000000000041240 rsi=0000000000041710 rdi=0000000000041240
rip=00000000750e9b59 rsp=00000000000411f0 rbp=000000000017f2a8
 r8=0000000000000001  r9=00000000750ffd40 r10=0000000000000000
r11=000000000017f29c r12=000000007efdb000 r13=000000000007fd20
r14=000000000007ee70 r15=00000000751e3380
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
wow64!Wow64PrepareForException+0×19:
00000000`750e9b59 654c8b1c2530000000 mov   r11,qword ptr gs:[30h] gs:00000000`00000030=????????????????

0:000> kc
Call Site
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher

It looks like a stack overflow. Usually it manifests via a PUSH instruction or a data access violation when ESP/RSP < TEB.StackLimit. However here RSP is still inside the stack range:

0:000> !teb
[...]
Wow64 TEB at 000000007efdb000
    ExceptionList:        000000007efdd000
    StackBase:            000000000007fd20
    StackLimit:           0000000000041000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efdb000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000001684 . 000000000000168c
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dq rsp-10 rsp+10
00000000`000411e0  00000000`00000000 00000000`00000000
00000000`000411f0  00000000`00000000 00000000`00000000
00000000`00041200  00000000`00000000

But still RSP is very close to the stack limit and the next function call surely would overflow because of the large frame delta:

0:000> kcf
  Memory  Call Site
          wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher
      5a0 wow64!Wow64PrepareForException
       50 ntdll!KiUserExceptionDispatcher

So we consider this as the stack overflow indeed. We specify the larger number of frames to reach the stack base:

0:000> kc 1000
Call Site
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
[...]
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64PrepareForException
ntdll!KiUserExceptionDispatcher
wow64!Wow64SystemServiceEx
wow64cpu!ServiceNoTurbo
wow64!RunCpuSimulation
wow64!Wow64LdrpInitialize
ntdll!LdrpInitializeProcess
ntdll! ?? ::FNODOBFM::`string’
ntdll!LdrInitializeThunk

We consider this as a nested unmanaged exception and try to see where it originated. First we double check that we don’t have any exceptions in 32-bit code. And indeed threads look clean:

0:000:x86> ~*kc

.  0  Id: 1684.168c Suspend: 0 Teb: 7efdb000 Unfrozen

user32!NtUserGetProp
user32!GetPropA
WARNING: Stack unwind information not available. Following frames may be wrong.
Application!foo
user32!InternalCallWinProc
user32!UserCallWinProcCheckWow
user32!SendMessageWorker
user32!SendMessageA
Application

   1  Id: 1684.1688 Suspend: 1 Teb: 7efd8000 Unfrozen

ntdll_77320000!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
rpcrt4!COMMON_ProcessCalls
rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents
rpcrt4!ProcessIOEventsWrapper
rpcrt4!BaseCachedThreadRoutine
rpcrt4!ThreadStartRoutine
kernel32!BaseThreadInitThunk
ntdll_77320000!__RtlUserThreadStart
ntdll_77320000!_RtlUserThreadStart

   2  Id: 1684.1678 Suspend: 1 Teb: 7efd5000 Unfrozen

ntdll_77320000!NtDelayExecution
kernel32!SleepEx
kernel32!Sleep
ole32!CROIDTable::WorkerThreadLoop
ole32!CRpcThread::WorkerLoop
ole32!CRpcThreadCache::RpcWorkerThreadEntry
kernel32!BaseThreadInitThunk
ntdll_77320000!__RtlUserThreadStart
ntdll_77320000!_RtlUserThreadStart

   3  Id: 1684.164c Suspend: 1 Teb: 7efad000 Unfrozen

ntdll_77320000!NtWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
msiltcfg!WorkerThread
kernel32!BaseThreadInitThunk
ntdll_77320000!__RtlUserThreadStart
ntdll_77320000!_RtlUserThreadStart

   4  Id: 1684.166c Suspend: 1 Teb: 7efaa000 Unfrozen

ntdll_77320000!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
winspool!MonitorRPCServerProcess
kernel32!BaseThreadInitThunk
ntdll_77320000!__RtlUserThreadStart
ntdll_77320000!_RtlUserThreadStart

The first thread looks a bit suspicious, we have never seen NtUserGetProp on stack traces, there is a possibility of an exception in main GUI thread and also the stack trace itself looks incorrect, suddenly sending a Windows message without any kind of a message loop:

0:000:x86> k
ChildEBP RetAddr 
0017f294 76e45be7 user32!NtUserGetProp+0×15
0017f2a8 025d4ba0 user32!GetPropA+0×3d
WARNING: Stack unwind information not available. Following frames may be wrong.
0017f2e0 76e38807 Application!foo+0×230
0017f30c 76e38962 user32!InternalCallWinProc+0×23
0017f384 76e3c4b6 user32!UserCallWinProcCheckWow+0×109
0017f3c8 76e3eae2 user32!SendMessageWorker+0×55b
0017f3ec 02a755c6 user32!SendMessageA+0×7f
00000000 00000000 Application+0×255c6

NtUserGetProp EIP address looks valid and points to the code after the return from a system call:

0:000:x86> u eip
user32!NtUserGetProp+0x15:
76e3b64f c20800          ret     8
76e3b652 90              nop
76e3b653 90              nop
76e3b654 90              nop
76e3b655 90              nop
76e3b656 90              nop
user32!GetPropW:
76e3b657 8bff            mov     edi,edi
76e3b659 55              push    ebp

0:000:x86> ub eip
user32!DefWindowProcW+0x96:
76e3b636 90              nop
76e3b637 90              nop
76e3b638 90              nop
76e3b639 90              nop
user32!NtUserGetProp:
76e3b63a b80e100000      mov     eax,100Eh
76e3b63f 8d542404        lea     edx,[esp+4]
76e3b643 b900000000      mov     ecx,0
76e3b648 64ff15c0000000  call    dword ptr fs:[0C0h]

Its 32-bit raw stack looks like normal main GUI thread of a VB6 application:

0:000:x86> !wow64exts.info
[...]
32 bit, StackBase   : 0×180000
        StackLimit  : 0×17c000
        Deallocation: 0×80000

64 bit, StackBase   : 0x7fd20
        StackLimit  : 0x41000
        Deallocation: 0x40000
[...]

0:000:x86> dds 0x17c000 0x180000
0017c000  00000000
0017c004  00000000
0017c008  00000000
0017c00c  00000000
0017c010  00000000
[...]
0017fdc8  00000000
0017fdcc  0017fe30
0017fdd0  76e38aad user32!DispatchMessageWorker+0×380
0017fdd4  00000000
0017fdd8  726ff5d1 msvbvm60!IID_IVbaHost+0×30ff1
0017fddc  0002088c
0017fde0  00000113
0017fde4  0002088c
0017fde8  00000000
0017fdec  00b319a8
0017fdf0  00000001
0017fdf4  9f7e168b
0017fdf8  76e395c0 user32!PeekMessageA
0017fdfc  02c0209c
0017fe00  00000000
0017fe04  76e395c0 user32!PeekMessageA
0017fe08  02c0216c
0017fe0c  0002088c
0017fe10  00000000
0017fe14  726aa76a msvbvm60!_vbaStrToAnsi+0×3ab
0017fe18  0017fdf4
0017fe1c  ffffffff
0017fe20  0017ff70
0017fe24  76e985e7 user32!_except_handler4
0017fe28  e98a626b
0017fe2c  fffffffe
0017fe30  0017fe40
0017fe34  76e39100 user32!DispatchMessageA+0xf
[…]
0017ff58  ffffffff
0017ff5c  ffffffff
0017ff60  ffffffff
0017ff64  00000000
0017ff68  0017ff10
0017ff6c  00000000
0017ff70  0017ffc4
0017ff74  7278bafd msvbvm60!CreateIExprSrvObj+0×9f2
0017ff78  726b97d0 msvbvm60!BASIC_CLASS_Release+0xadbd
0017ff7c  00000000
0017ff80  0017ff94
0017ff84  00401396
0017ff88  00401514
0017ff8c  768de3f3 kernel32!BaseThreadInitThunk+0xe
0017ff90  7efde000
0017ff94  0017ffd4
0017ff98  7739cfed ntdll_77320000!__RtlUserThreadStart+0×23
0017ff9c  7efde000
0017ffa0  63553bf6
0017ffa4  00000000
0017ffa8  00000000
0017ffac  7efde000
0017ffb0  00000000
0017ffb4  00000000
0017ffb8  00000000
0017ffbc  0017ffa0
0017ffc0  00000000
0017ffc4  0017ffe4
0017ffc8  773d2926 ntdll_77320000!_except_handler4
0017ffcc  14761e1a
0017ffd0  00000000
0017ffd4  0017ffec
0017ffd8  7739d1ff ntdll_77320000!_RtlUserThreadStart+0×1b
0017ffdc  0040138c
0017ffe0  7efde000
0017ffe4  ffffffff
0017ffe8  773bd377 ntdll_77320000!FinalExceptionHandler
0017ffec  00000000
0017fff0  00000000
0017fff4  0040138c
0017fff8  7efde000
0017fffc  00000000
00180000  78746341

There are no any signs of exception codes and processing residue there and we come back to out 64-bit layer:

0:000:x86> .effmach amd64
Effective machine: x64 (AMD64)

We dump 64-bit stack to see the moment when the first exception happened:

0:000> dqs 0x41000 0x7fd20
[...]
00000000`0007df30  00000000`00000000
00000000`0007df38  00000000`00000000
00000000`0007df40  00000000`00000000
00000000`0007df48  00000000`00000000
00000000`0007df50  00000000`00000000
00000000`0007df58  00000000`00000000
00000000`0007df60  00000000`00000000
00000000`0007df68  00000000`00000000
00000000`0007df70  00000000`00000000
00000000`0007df78  00000000`00000000
00000000`0007df80  00000000`750e9b59 wow64!Wow64PrepareForException+0x19
00000000`0007df88  00000000`00000000
00000000`0007df90  00000000`00000000
00000000`0007df98  00000000`0007dfb0
00000000`0007dfa0  00000000`00000001
00000000`0007dfa8  00000000`0007e510
00000000`0007dfb0  00000000`611b422e
00000000`0007dfb8  fffffa60`04eec978
00000000`0007dfc0  fffffa60`04ee6520
00000000`0007dfc8  fffffa60`04ee6520
00000000`0007dfd0  fffffa60`04eec978
00000000`0007dfd8  00000000`00000030
00000000`0007dfe0  00000000`0017f29c
00000000`0007dfe8  00000000`0000c04c
00000000`0007dff0  00000000`00000000
00000000`0007dff8  00000000`771c59e6 ntdll!KiUserExceptionDispatcher+0×1c
00000000`0007e000  01c9f89c`fe787c8d
00000000`0007e008  fffffa60`03732e4e
00000000`0007e010  fffffa80`0497c9a0
00000000`0007e018  00000000`00000000
00000000`0007e020  00000000`00000001
00000000`0007e028  01c9f89c`fe787c00
00000000`0007e030  00001fa0`0010001f
00000000`0007e038  0053002b`002b0033
00000000`0007e040  00010283`002b002b
00000000`0007e048  00000000`00000000
00000000`0007e050  00000000`00000000
00000000`0007e058  00000000`00000000
00000000`0007e060  00000000`00000000
00000000`0007e068  00000000`00000000
00000000`0007e070  00000000`00000000
00000000`0007e078  00000000`00000006
00000000`0007e080  00000000`0000100e
00000000`0007e088  00000000`0000000e
00000000`0007e090  00000000`0017f29c
00000000`0007e098  00000000`0007e5a0
00000000`0007e0a0  00000000`0017f2a8
00000000`0007e0a8  00000000`0000c04c
00000000`0007e0b0  00000000`00000000
00000000`0007e0b8  00000000`00000001
00000000`0007e0c0  00000000`750ffd40 wow64!ServiceTables
00000000`0007e0c8  00000000`00000000
00000000`0007e0d0  00000000`0017f29c
00000000`0007e0d8  00000000`7efdb000
00000000`0007e0e0  00000000`0007fd20
00000000`0007e0e8  00000000`0007ee70
00000000`0007e0f0  00000000`751e3380 wow64cpu!CpupSaveLegacyFloatingPointState+0×60
00000000`0007e0f8  00000000`750ca923 wow64!Wow64SystemServiceEx+0×57
00000000`0007e100  015c0000`4000137f
00000000`0007e108  00000023`02add97f
00000000`0007e110  0000002b`0017f408
00000000`0007e118  0000ffff`00001f80
00000000`0007e120  00000000`00000000
00000000`0007e128  00000000`00000000

All addresses we used to try for .exr and .cxr commands in hidden exception pattern are beyond user space and we therefore conclude that somehow such structures or pointers to them became corrupt and their access triggered the same exception processing code over and over again finally leading to the stack overflow. This was an x64 system and taking a complete memory dump was impractical so at the same time when a WER error message appeared a kernel memory dump was generated (a pair of dumps from fiber bundle) and it shows the final stages of exception processing that are in agreement with the user dump we analyzed.

Initially we thought that the system experienced a bugcheck after the application crash:

1: kd> !analyze -v
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high.  This is usually caused by drivers using improper addresses. If kernel debugger is available get stack backtrace.
Arguments:
Arg1: fffff880058b1010, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffffa60053da17a, address which referenced memory

But then we noticed that bugcheck was forced intentionally using NotMyFault sysinternals tool:

1: kd> kc
Call Site
nt!KeBugCheckEx
nt!KiBugCheckDispatch
nt!KiPageFault
myfault
myfault
nt!IopXxxControlFile
nt!NtDeviceIoControlFile
nt!KiSystemServiceCopyEnd
0×0

In the kernel dump we see our application and the next process is WER: 

1: kd> !vm
[...]
PROCESS fffffa8001886710
    SessionId: 1  Cid: 13e4    Peb: 7efdf000  ParentCid: 0ab0
    DirBase: 329c8000  ObjectTable: fffff880059424d0  HandleCount: 190.
    Image: Application.exe

PROCESS fffffa800188a040
    SessionId: 1  Cid: 0b44    Peb: 7efdf000  ParentCid: 0934
    DirBase: 2b144000  ObjectTable: fffff88004c68a30  HandleCount: 166.
    Image: WerFault.exe

We see a page fault on a kernel stack trace of the main process thread:

1: kd> !thread fffffa8001cc7bb0 1f
THREAD fffffa8001cc7bb0  Cid 13e4.01dc  Teb: 000000007efdb000 Win32Thread: fffff900c210ed50 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa8001cc7f40  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff880075c3970 : queued at port fffffa8001dae290 : owned by process fffffa8001d8e040
Not impersonating
DeviceMap                 fffff88004c30a00
Owning Process            fffffa8001886710       Image:         Application.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      83669865       Ticks: 320 (0:00:00:04.992)
Context Switch Count      1829                 LargeStack
UserTime                  00:00:00.187
KernelTime                00:00:00.280
Win32 Start Address 0x000000000040138c
Stack Init fffffa6004cc3db0 Current fffffa6004cc2ae0
Base fffffa6004cc4000 Limit fffffa6004cb9000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 2 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`04cc2b20 fffff800`0165cf8a nt!KiSwapContext+0×7f
fffffa60`04cc2c60 fffff800`0165e38a nt!KiSwapThread+0×2fa
fffffa60`04cc2cd0 fffff800`0168ecab nt!KeWaitForSingleObject+0×2da
fffffa60`04cc2d60 fffff800`018e1f64 nt!AlpcpSignalAndWait+0×7b
fffffa60`04cc2da0 fffff800`018e7216 nt!AlpcpReceiveSynchronousReply+0×44
fffffa60`04cc2e00 fffff800`018dc27f nt!AlpcpProcessSynchronousRequest+0×24f
fffffa60`04cc2f20 fffff800`01656e33 nt!NtAlpcSendWaitReceivePort+0×19f
fffffa60`04cc2fd0 fffff800`01657340 nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`04cc3040)
fffffa60`04cc31d8 fffff800`019b6e30 nt!KiServiceLinkage
fffffa60`04cc31e0 fffff800`01898d6d nt!DbgkpSendErrorMessage+0×230
fffffa60`04cc3300 fffff800`016bf487 nt!DbgkForwardException+0×12d
fffffa60`04cc3440 fffff800`016571e9 nt! ?? ::FNODOBFM::`string’+0×29550
fffffa60`04cc3a40 fffff800`01655fe5 nt!KiExceptionDispatch+0xa9
fffffa60`04cc3c20 00000000`75319b59 nt!KiPageFault+0×1e5 (TrapFrame @ fffffa60`04cc3c20)
00000000`000411f0 00000000`00000000 0×75319b59

Analysis of the raw stack confirms the final access violation address:

1: kd> dqs fffffa6004cb9000 fffffa6004cc4000
[…]
fffffa60`04cc3a00  00000000`00000000
fffffa60`04cc3a08  00000000`00000000
fffffa60`04cc3a10  00000000`00000000
fffffa60`04cc3a18  00000000`00000000
fffffa60`04cc3a20  00000000`0007fd20
fffffa60`04cc3a28  00000000`7efdb000
fffffa60`04cc3a30  00000000`00041240
fffffa60`04cc3a38  fffff800`016571e9 nt!KiExceptionDispatch+0xa9
fffffa60`04cc3a40  fffffa60`04cc3b78
fffffa60`04cc3a48  00000000`0017f290
fffffa60`04cc3a50  fffffa60`04cc3c20
fffffa60`04cc3a58  00000000`00041710
fffffa60`04cc3a60  00000000`00000001
fffffa60`04cc3a68  00000000`80000001
fffffa60`04cc3a70  0000002d`0000002d
fffffa60`04cc3a78  0000002d`0000002d
fffffa60`04cc3a80  0000002d`0000002d
fffffa60`04cc3a88  0000002d`0000002d
fffffa60`04cc3a90  00000000`00000000
fffffa60`04cc3a98  00000000`00000000
fffffa60`04cc3aa0  00000000`00000000
fffffa60`04cc3aa8  00000000`00000000
fffffa60`04cc3ab0  00000000`00000000
fffffa60`04cc3ab8  00000000`00000000
fffffa60`04cc3ac0  00000000`00000000
fffffa60`04cc3ac8  00000000`00000000
[…]

1: kd> .exr fffffa60`04cc3b78
ExceptionAddress: 0000000075319b59
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000030
Attempt to read from address 0000000000000030

Looking at drivers shows that the system was virtualized under a VM:

1: kd> lm
[...]
fffffa60`037bc000 fffffa60`037e0000   3rdpartyVM   (deferred)     
[...]

Therefore we suggested to test the application without VM to rule out vitrualization influence.

- Dmitry Vostokov @ DumpAnalysis.org -

When complete memory dumps are huge (in case of x64 systems) we can dump specific processes and then force a kernel memory dump. Here we have a product of spaces similar to a fiber bundle illustrated by the following intuitive picture:

- Dmitry Vostokov @ DumpAnalysis.org -

The second issue of the magazine was put into production today and should be available after one or two weeks on Amazon, B&N and other booksellers worldwide.

Title: Debugged! MZ/PE: Modeling Software Defects
Authors: Konstantin Chebotarev, Kapildev Ramlal, Dmitry Vostokov
ISBN: 1906717680
ISBN-13: 978-1906717681
Annotation: Welcome to the second issue of Debugged! MZ/PE magazine! It brings fault injection into new light and features articles discussing software defect construction via DLL injection, modeling CPU spikes and runaway exception processing. This issue also includes a memory dump analysis certification voucher. Back cover features WinDbg breakpoint and tracing command summary for easy desk reference. - Dmitry Vostokov - Editor-in-Chief

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we get rare or hardware-related bugchecks and these might have been influenced by hardware virtualization (that is also a software and could have its own defects). Therefore it is beneficial to recognize when a system is running under a VM (Virtualized System pattern):

Memory Dumps from Xen-virtualized Windows

Memory dumps from VMware images (Virtual PC diagnostics in post comments)

Memory Dumps from Hyper-Virtualized Windows

For example, we get the following bugcheck and stack trace for the first processor:

0: kd> !analyze -v
[...]
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 00000060, Clock interrupt time out interval in nominal clock ticks.
Arg2: 00000000, 0.
Arg3: 805d8120, The PRCB address of the hung processor.
Arg4: 00000001, 0.

CURRENT_IRQL:  1c

STACK_TEXT: 
8d283694 816df8a5 nt!KeBugCheckEx+0x1e
8d2836c8 816e163d nt!KeUpdateRunTime+0xd5
8d2836c8 84617008 nt!KeUpdateSystemTime+0xed
WARNING: Frame IP not in any known module. Following frames may be wrong.
8d283748 816f46a1 0×84617008
8d283758 816fa6aa nt!HvlpInitiateHypercall+0×21
8d283784 8166aca5 nt!HvlNotifyLongSpinWait+0×2b
8d2837a0 816cce7e nt!KeFlushSingleTb+0xc4
8d283808 81681db4 nt!MmAccessFault+0xc1d
8d283808 816dd033 nt!KiTrap0E+0xdc
8d28389c 8168ed58 nt!memcpy+0×33
8d283954 816712bf nt!MmCopyToCachedPage+0×1193
8d2839ec 81663053 nt!CcMapAndCopy+0×210
8d283a74 8c688218 nt!CcFastCopyWrite+0×283
8d283b98 8c40badc Ntfs!NtfsCopyWriteA+0×23e
8d283bcc 8c40bcab fltmgr!FltpPerformFastIoCall+0×22e
8d283bf8 8c41dc30 fltmgr!FltpPassThroughFastIo+0×7d
8d283c3c 818471cd fltmgr!FltpFastIoWrite+0×146
8d283d38 8167ec7a nt!NtWriteFile+0×34c
8d283d38 77115e74 nt!KiFastCallEntry+0×12a
01cfee80 00000000 0×77115e74

The thread was servicing a page fault. Notice the gap between KeUpdateSystemTime and HvlpInitiateHypercall. This is normal and consistent code if we look closer:

0: kd> .asm no_code_bytes
Assembly options: no_code_bytes

0: kd> uf HvlpInitiateHypercall
nt!HvlpInitiateHypercall:
816f4680 push    edi
816f4681 push    esi
816f4682 push    ebx
816f4683 mov     eax,dword ptr [esp+10h]
816f4687 mov     edx,dword ptr [esp+14h]
816f468b mov     ecx,dword ptr [esp+18h]
816f468f mov     ebx,dword ptr [esp+1Ch]
816f4693 mov     esi,dword ptr [esp+20h]
816f4697 mov     edi,dword ptr [esp+24h]
816f469b call    dword ptr [nt!HvlpHypercallCodeVa (8176bb8c)]
816f46a1 pop     ebx
816f46a2 pop     esi
816f46a3 pop     edi
816f46a4 ret     18h

0: kd> dp 8176bb8c l1
8176bb8c  84617000

0: kd> uf 84617000
84617000 or      eax,80000000h
84617005 vmcall
84617008ret

We have the address of RET instruction (84617008) on the stack trace:

0: kd> kv
ChildEBP RetAddr  Args to Child             
8d283694 816df8a5 00000101 00000060 00000000 nt!KeBugCheckEx+0x1e
8d2836c8 816e163d 84e1521b 000000d1 8d283784 nt!KeUpdateRunTime+0xd5
8d2836c8 8461700884e1521b 000000d1 8d283784 nt!KeUpdateSystemTime+0xed (FPO: [0,2] TrapFrame @ 8d2836d8)
WARNING: Frame IP not in any known module. Following frames may be wrong.
8d283748 816f46a1 84e6c900 22728000 8172e28c 0×84617008
8d283758 816fa6aa 00010008 00000000 22728000 nt!HvlpInitiateHypercall+0×21 (FPO: [6,3,0])
8d283784 8166aca5 22728000 00000000 00000000 nt!HvlNotifyLongSpinWait+0×2b
[…]

The second processor is busy too:

0: kd> !running

System Processors 3 (affinity mask)
  Idle Processors 0

Prcbs  Current   Next   
  0    8172c920  84e6c900            ................
  1    805d8120  85138030  85a50d78  …………….

0: kd> !thread 85138030
THREAD 85138030  Cid 0564.11c8  Teb: 7ff9f000 Win32Thread: 00000000 RUNNING on processor 1
IRP List:
    85ab5d00: (0006,01fc) Flags: 00000884  Mdl: 00000000
    85445ab8: (0006,0094) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 98a7d558
Owning Process            84f0d938       Image:         Application.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      695643         Ticks: 224 (0:00:00:03.500)
Context Switch Count      20            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x705e2679
Stack Init a1d13000 Current a1d10a70 Base a1d13000 Limit a1d10000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr  Args to Child             
00000000 00000000 00000000 00000000 00000000 0×0

Because we have (Limit, Current, Base) triple for our thread we check its execution residue on kernel raw stack. We find traces of a hypercall too:

0: kd> dds a1d10000 a1d13000
[...]
a1d1215c  816fa6da nt!HvlEndSystemInterrupt+0x20
a1d12160  40000070
a1d12164  00000000
a1d12168  81608838 hal!HalEndSystemInterrupt+0x7a
a1d1216c  805d8000
a1d12170  a1d12180
a1d12174  81618cc9 hal!HalpIpiHandler+0x189
a1d12178  84f4841b
a1d1217c  000000e1
a1d12180  a1d1222c
a1d12184  84617008
a1d12188  badb0d00
a1d1218c  00000000
a1d12190  81665699 nt!RtlWalkFrameChain+0x58
a1d12194  816659c4 nt!RtlWalkFrameChain+0x377
a1d12198  11c4649e
a1d1219c  00000002
a1d121a0  00000003
a1d121a4  85478444
a1d121a8  0000001f
a1d121ac  00000000
a1d121b0  00000000
a1d121b4  816f46a1 nt!HvlpInitiateHypercall+0×21
a1d121b8  8172c800 nt!KiInitialPCR
a1d121bc  85138030
a1d121c0  85a72ac0
a1d121c4  8171b437 nt!HvlSwitchVirtualAddressSpace+0×28
a1d121c8  00010001
a1d121cc  00000000
a1d121d0  85a72ac0
a1d121d4  85138030
a1d121d8  8172c802 nt!KiInitialPCR+0×2
a1d121dc  00000000
a1d121e0  85138030
a1d121e4  816fa6da nt!HvlEndSystemInterrupt+0×20
a1d121e8  40000070
a1d121ec  00000000
a1d121f0  81608838 hal!HalEndSystemInterrupt+0×7a
a1d121f4  816f46a1 nt!HvlpInitiateHypercall+0×21
a1d121f8  805d8000
a1d121fc  85138030
a1d12200  805dc1e0
a1d12204  8171b437 nt!HvlSwitchVirtualAddressSpace+0×28
a1d12208  00010001
a1d1220c  00000000
a1d12210  00000000
a1d12214  81608468 hal!HalpDispatchSoftwareInterrupt+0×5e
a1d12218  00000000
a1d1221c  00000000
a1d12220  00000206
a1d12224  a1d12240
a1d12228  81608668 hal!HalpCheckForSoftwareInterrupt+0×64
a1d1222c  00000002
a1d12230  00000000
a1d12234  816086a8 hal!KfLowerIrql
a1d12238  00000000
a1d1223c  00000002
a1d12240  a1d12250
a1d12244  8160870c hal!KfLowerIrql+0×64
a1d12248  00000000
a1d1224c  00000000
a1d12250  a1d12294
a1d12254  816e035a nt!KiSwapThread+0×477
a1d12258  85138030
a1d1225c  851380b8
a1d12260  805d8120
a1d12264  0014d1f8
[…]

Looking at raw stack further we can even see that it was processing a page fault too and manually reconstruct its stack trace:

[...]
a1d1074c  85aef510
a1d10750  a1d10768
a1d10754  81678976 nt!IofCallDriver+0×63
a1d10758  84c87d50
a1d1075c  85aef510
a1d10760  00000000
a1d10764  84c87d50
a1d10768  a1d10784
a1d1076c  8166d74e nt!IoPageRead+0×172
a1d10770  85138030
a1d10774  84a1352c
a1d10778  84a134f8
a1d1077c  84a13538
a1d10780  84c87d50
a1d10784  a1d10840
a1d10788  816abf07 nt!MiDispatchFault+0xd14
a1d1078c  00000043
a1d10790  85138030
a1d10794  84a13538
a1d10798  84a1350c
a1d1079c  84a1352c
a1d107a0  8174c800 nt!MmSystemCacheWs
a1d107a4  00000000
a1d107a8  85138030
a1d107ac  a5397bf8
a1d107b0  85b01c48
a1d107b4  00000000
a1d107b8  00000000
a1d107bc  a5397bf8
a1d107c0  84a1358c
a1d107c4  a1d10864
a1d107c8  00000000
a1d107cc  8463a590
a1d107d0  84a134f8
a1d107d4  c0518000
a1d107d8  00000000
a1d107dc  00000000
a1d107e0  00000028
a1d107e4  a1d107f4
a1d107e8  00000000
a1d107ec  00000038
a1d107f0  859f5640
a1d107f4  a4bfa390
a1d107f8  00000000
a1d107fc  00000000
a1d10800  00000000
a1d10804  a1d10938
a1d10808  818652bd nt!MmCreateSection+0×98f
a1d1080c  00000000
a1d10810  846652e8
a1d10814  00000000
a1d10818  00000000
a1d1081c  00000000
a1d10820  00000028
a1d10824  00000000
a1d10828  00000080
a1d1082c  0000000a
a1d10830  85ae1c98
a1d10834  85ae1c20
a1d10838  00000000
a1d1083c  00000000
a1d10840  a1d108b8
a1d10844  816cd325 nt!MmAccessFault+0×10c6
a1d10848  a3000000
a1d1084c  a5397bf8
a1d10850  00000000
a1d10854  8174c800 nt!MmSystemCacheWs
a1d10858  00000000
a1d1085c  00000000
a1d10860  a5397bf8
a1d10864  00000000
[…]

0: kd> k L=a1d10750 a1d10750 a1d10750
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
a1d10750 81678976 0xa1d10750
a1d10768 8166d74e nt!IofCallDriver+0x63
a1d10784 816abf07 nt!IoPageRead+0x172
a1d10840 816cd325 nt!MiDispatchFault+0xd14
a1d108b8 816f0957 nt!MmAccessFault+0x10c6
a1d10924 8181c952 nt!MmCheckCachedPageState+0x801
a1d109b0 8c60f850 nt!CcCopyRead+0x435
a1d109dc 8c613c52 Ntfs!NtfsCachedRead+0x13b
a1d10abc 8c612b6f Ntfs!NtfsCommonRead+0x105a
a1d10b2c 81678976 Ntfs!NtfsFsdRead+0x273
a1d10b44 8c40cba7 nt!IofCallDriver+0x63
a1d10b68 8c40d7c7 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x251
a1d10ba0 8c40dbe7 fltmgr!FltPerformSynchronousIo+0xb9
a1d10c10 9ca680e5 fltmgr!FltReadFile+0x2ed
[...]

A bit of reverse engineering offtopic here: I think that the absence of a trap means that cache processing NTFS code reuses page fault handling code.

- Dmitry Vostokov @ DumpAnalysis.org -

The common way to do memory analysis is through a powerful agency. Then certain sequences of actions become a common ritual practice. Here is my definition of ritual in this context based on Victor Turner’s cited in Religion: The Basics:

Formal actions that rely on beliefs in perceived powers.

Typical example from Windows crash dump analysis includes WinDbg as an agency and typing !analyze -v command.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we follow the course of established troubleshooting or debugging methods without questioning them or not considering possible exceptions. For example, the usual advice to heap corruption signs in process memory dumps is to ask to enable full page heap. However page heap helps to catch buffer overwrites but not underwrites and heap can also be damaged by other means like double free, passing an invalid address or direct corruption of control structures via a dangling pointer. Therefore in some cases when we enable full page heap we get the same stack trace:

Normal:

0:030> kL 100
ChildEBP RetAddr 
175af77c 7d620ec5 ntdll_7d600000!RtlpCoalesceFreeBlocks+0×231
175af864 7c34218a ntdll_7d600000!RtlFreeHeap+0×38e
175af8ac 67d6153d msvcr71!free+0xc3
175af950 67781d69 Component!OnData+0×504
175af96c 7da4d03d Component!DispatchRequest+0×99
175afc28 7da4d177 rpcrt4!DispatchToStubInCNoAvrf+0×38
175afc7c 7da4d812 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
175afca0 7da59a1c rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
175afcfc 7da4da21 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×421
175afd20 7da3db14 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
175aff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
175aff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd
175affac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0×9d
175affb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0×1b
175affec 00000000 kernel32!BaseThreadStart+0×34

With full page heap enabled:

0:030> kL
ChildEBP RetAddr 
6092f568 7d6822fa ntdll!RtlpDphIsNormalHeapBlock+0×1f
6092f598 7d68256c ntdll!RtlpDphNormalHeapFree+0×21
6092f5f0 7d685443 ntdll!RtlpDebugPageHeapFree+0×146
6092f658 7d65714a ntdll!RtlDebugFreeHeap+0×2c
6092f730 7d62c5c0 ntdll!RtlFreeHeapSlowly+0×37
6092f814 7c34218a ntdll!RtlFreeHeap+0×11a
6092f85c 67d6153d msvcr71!free+0xc3
6092f900 67781d69 Component!OnData+0×504
6092f91c 7da4d03d Component!DispatchRequest+0×99
6092fc10 7da7ec5e rpcrt4!DispatchToStubInCNoAvrf+0×38
6092fc28 7da4d177 rpcrt4!DispatchToStubInCAvrf+0×14
6092fc7c 7da4d812 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
6092fca0 7da59a1c rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
6092fcfc 7da4da21 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×421
6092fd20 7da3db14 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
6092ff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
6092ff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd
6092ffac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0×9d
6092ffb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0×1b
6092ffec 00000000 kernel32!BaseThreadStart+0×34

We see that we get the same Component stack trace before calling free (shown in blue above) so we better search in a database of stack trace signatures before sending Habitual Reply to enable page heap. If there is a match in database then the course of future actions can be completely different: an immediate available hotfix or we find a similar problem that is already under investigation. The sooner we provide a relief to our customer the better.

- Dmitry Vostokov @ DumpAnalysis.org -

It happened to me that I accidentally opened a YouTube page and it started playing video so I tried to either stop it or pushed the back browser button. Then immediately WER dialog appeared ruining my intricate multi-tab window. Nevertheless I looked at this event from the positive perspective as an opportunity to analyze a dump file and write a case study. I saved the crash dump using Windows Server 2008 Task Manager for later postmortem analysis.

Today I had a chance to look at it. When we open the dump file we see that it is the dump of a WOW64 process:

0:000> !analyze -v

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

STACK_TEXT: 
00000000`0016ed78 00000000`75fcab46 : 00000000`77ddc990 00000000`0016fd20 00000000`75efa2b1 00000000`0016f760 : wow64cpu!WaitForMultipleObjects32+0×3a
00000000`0016ee20 00000000`75fca14c : 00000000`00000000 00000000`00000000 00000000`75fc3258 00000000`7ffe0030 : wow64!RunCpuSimulation+0xa
00000000`0016ee50 00000000`77d052d3 : 00000000`77cd0000 00000000`00000000 00000000`77de61b0 00000000`77cd0000 : wow64!Wow64LdrpInitialize+0×4b4
00000000`0016f3b0 00000000`77d05363 : 00000000`00000000 00000000`00000000 00000000`7efdf000 00000000`77d07ca7 : ntdll!LdrpInitializeProcess+0×14ac
00000000`0016f660 00000000`77cf85ce : 00000000`0016f760 00000000`00000000 00000000`7efdf000 00000000`00000000 : ntdll! ?? ::FNODOBFM::`string’+0×1ff19
00000000`0016f710 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

FOLLOWUP_IP:
wow64cpu!WaitForMultipleObjects32+3a
00000000`75ec374f 418bbda0000000  mov     edi,dword ptr [r13+0A0h]

MODULE_NAME: wow64cpu 

Then we load a WOW64 extension that allows us to analyze 32-bit parts of saved 64-bit dumps and we retry the default analysis:

0:000> .load wow64exts; .effmach x86
Effective machine: x86 compatible (x86)

0:000:x86> !analyze -v

IP_ON_HEAP:  ffffffffc0d84c02
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

STACK_TEXT: 
00000000`003b0068 iexplore!__dyn_tls_init_callback <PERF> +0×0
ffffffff`c0d84c02 unknown+0×0
ffffffff`aacf27dc unknown+0×0
ffffffff`c1d84c02 unknown+0×0
ffffffff`b3cf38dc unknown+0×0
ffffffff`c2d84c02 unknown+0×0
00000000`3ecf38dc unknown+0×0
[…]
00000000`7aca19dd unknown+0×0
ffffffff`e0d84c02 unknown+0×0
ffffffff`a7ca19dd unknown+0×0
ffffffff`

FOLLOWUP_IP:
iexplore!__dyn_tls_init_callback <PERF> (iexplore+0x80068)+0
003b0068 70f7            jo      iexplore!__dyn_tls_init_callback <PERF> (iexplore+0x80061) (003b0061)

Clearly the shown above stack trace is incorrect but at the same time the first thread stack looks consistent:

0:000:x86> k 100
ChildEBP RetAddr 
0051d930 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15
0051d9cc 77888f76 kernel32!WaitForMultipleObjectsEx+0x11d
0051da20 7617ab28 user32!RealMsgWaitForMultipleObjectsEx+0x14d
0051da48 7617ac88 ole32!CCliModalLoop::BlockFn+0x97
0051da70 7617ad74 ole32!ModalLoop+0x5b
0051da80 76287c28 ole32!SwitchSTA+0x21
0051daac 762889d4 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x127
0051db8c 7617ad2e ole32!CRpcChannelBuffer::SendReceive2+0xef
0051dba8 7617ace0 ole32!CCliModalLoop::SendReceive+0x1e
0051dc20 7619e688 ole32!CAptRpcChnl::SendReceive+0x73
0051dc74 779f1074 ole32!CCtxComChnl::SendReceive+0x1c5
0051dc90 779f102b rpcrt4!NdrProxySendReceive+0x49
0051dc9c 779f0146 rpcrt4!NdrpProxySendReceive+0xb
0051e0a4 779f11ee rpcrt4!NdrClientCall2+0x18f
0051e0c8 779649c2 rpcrt4!ObjectStublessClient+0x90
0051e0d8 74bc86e3 rpcrt4!ObjectStubless+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0051e0e4 74bda9bb ieframe+0x1986e3
0051e0f0 74b020d9 ieframe+0x1aa9bb
0051e164 74adbb6a ieframe+0xd20d9
0051e18c 77888807 ieframe+0xabb6a
0051e1b8 77888962 user32!InternalCallWinProc+0x23
0051e230 7788c4b6 user32!UserCallWinProcCheckWow+0x109
0051e274 7788c517 user32!SendMessageWorker+0x55b
0051e298 75b3f74a user32!SendMessageW+0x7f
0051e2c0 75b3f63b comctl32!CToolbar::TBOnLButtonUp+0x12f
0051e368 75b37020 comctl32!CToolbar::ToolbarWndProc+0xaa9
0051e388 77888807 comctl32!CToolbar::s_ToolbarWndProc+0x9b
0051e3b4 77888962 user32!InternalCallWinProc+0x23
0051e42c 77888aad user32!UserCallWinProcCheckWow+0x109
0051e490 77888b00 user32!DispatchMessageWorker+0x380
0051e4a0 74ae07ce user32!DispatchMessageW+0xf
0051e4d4 74a7ffce ieframe+0xb07ce
0051e4e0 74a6f579 ieframe+0x4ffce
0051e504 74a6f4c7 ieframe+0x3f579
0051f574 74a5d1ba ieframe+0x3f4c7
0051f7a4 003333c3 sxs!_hmod__RPCRT4_dll <PERF> (sxs+0x8d1ba)
0051fbe8 0033325a iexplore!wWinMain+0x27b
0051fc7c 775de4a5 iexplore!_initterm_e+0x1b1
0051fc88 77eecfed kernel32!BaseThreadInitThunk+0xe
0051fcc8 77eed1ff ntdll_77e70000!__RtlUserThreadStart+0x23
0051fce0 00000000 ntdll_77e70000!_RtlUserThreadStart+0x1b

We look at the stack trace collection to find another exception and we see it indeed on 7th stack trace:

0:000:x86> ~*k 100

[...]

   7  Id: b44.b1c Suspend: 0 Teb: 7efad000 Unfrozen
ChildEBP RetAddr 
05baee9c 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baef38 775649d9 kernel32!WaitForMultipleObjectsEx+0x11d
05baef54 7761573d kernel32!WaitForMultipleObjects+0x18
05baefc0 77615969 kernel32!WerpReportFaultInternal+0x16d
05baefd4 775ec66f kernel32!WerpReportFault+0×70
05baf060 77eed03e kernel32!UnhandledExceptionFilter+0×1b5
05baf068 77ebf2d0 ntdll_77e70000!__RtlUserThreadStart+0×6f
05baf07c 77f229b3 ntdll_77e70000!_EH4_CallFilterFunc+0×12
05baf0a4 77e93099 ntdll_77e70000!_except_handler4+0×8e
05baf0c8 77e9306b ntdll_77e70000!ExecuteHandler2+0×26
05baf178 77e92eff ntdll_77e70000!ExecuteHandler+0×24
05baf198 7757f328 ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf4fc 7155dead kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d Flash10b!DllUnregisterServer+0×1adbe0
05baf550 7150ca25 Flash10b!DllUnregisterServer+0×1aa2d0
05baf564 7150b5b5 Flash10b!DllUnregisterServer+0×15c758
05baf588 7150b90a Flash10b!DllUnregisterServer+0×15b2e8
00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

[...]

  97  Id: b44.ff4 Suspend: 1 Teb: 7ee9f000 Unfrozen
ChildEBP RetAddr 
7387facc 77561270 ntdll_77e70000!ZwWaitForSingleObject+0x15
7387fb3c 775611d8 kernel32!WaitForSingleObjectEx+0xbe
7387fb50 713e0a11 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
7387fb74 713e0d34 Flash10b!DllUnregisterServer+0x30744
7387fb78 775de4a5 Flash10b!DllUnregisterServer+0x30a67
7387fb84 77eecfed kernel32!BaseThreadInitThunk+0xe
7387fbc4 77eed1ff ntdll_77e70000!__RtlUserThreadStart+0x23
7387fbdc 00000000 ntdll_77e70000!_RtlUserThreadStart+0x1b

Then we switch to this thread and try to find and set the new exception context:

0:000:x86> ~7s
ntdll_77e70000!NtWaitForMultipleObjects+0x15:
77e90bc5 c21400          ret     14h

0:007:x86> kv
ChildEBP RetAddr  Args to Child             
05baee9c 7756e91a 00000002 05baeeec 00000001 ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baef38 775649d9 05baeeec 05baef88 00000000 kernel32!WaitForMultipleObjectsEx+0x11d
05baef54 7761573d 00000002 05baef88 00000000 kernel32!WaitForMultipleObjects+0x18
05baefc0 77615969 05baf090 00000001 00000001 kernel32!WerpReportFaultInternal+0x16d
05baefd4 775ec66f 05baf090 00000001 19e00de6 kernel32!WerpReportFault+0x70
05baf060 77eed03e 00000000 77ebf2d0 00000000 kernel32!UnhandledExceptionFilter+0×1b5
05baf068 77ebf2d0 00000000 05bafacc 77e9da38 ntdll_77e70000!__RtlUserThreadStart+0×6f
05baf07c 77f229b3 00000000 00000000 00000000 ntdll_77e70000!_EH4_CallFilterFunc+0×12
05baf0a4 77e93099 fffffffe 05bafabc 05baf1e0 ntdll_77e70000!_except_handler4+0×8e
05baf0c8 77e9306b 05baf190 05bafabc 05baf1e0 ntdll_77e70000!ExecuteHandler2+0×26
05baf178 77e92eff 01baf190 05baf1e0 05baf190 ntdll_77e70000!ExecuteHandler+0×24
05baf198 7757f328 00000003 19930520 05baf544 ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf4fc 7155dead e06d7363 00000001 00000003 kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d 05baf544 715e7954 715e5ac0 Flash10b!DllUnregisterServer+0×1adbe0
05baf550 7150ca25 002c1f7c 4237d4f8 4237c598 Flash10b!DllUnregisterServer+0×1aa2d0
05baf564 7150b5b5 002c1f60 00000010 423d0b28 Flash10b!DllUnregisterServer+0×15c758
05baf588 7150b90a 4237cb38 7150ba0b 00000000 Flash10b!DllUnregisterServer+0×15b2e8
00000000 00000000 00000000 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

Unfortunately a pointer to an exception pointers structure is NULL (probably because of optimized code) and we cannot use .exptr command. However we can still use a technique described in Hidden Exception pattern using 32-bit raw stack to set the context and display the exception record:

0:007:x86> !wow64exts.info

PEB32: 0x7efde000
PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0x7efaf000
TEB64: 0x7efad000

32 bit, StackBase   : 0×5bb0000
        StackLimit  : 0×5b95000
        Deallocation: 0×5ab0000

64 bit, StackBase   : 0x30ffd20
        StackLimit  : 0x30f8000
        Deallocation: 0x30c0000

0:007:x86> dps 0×5b95000 0×5bb0000
05b95000  00000000
05b95004  00000000
05b95008  00000000
[…]
05baf160  00000009
05baf164  00000000
05baf168  00000000
05baf16c  05bb0000
05baf170  05b95000
05baf174  00000000
05baf178  05baf4fc
05baf17c  77e92eff ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf180  01baf190
05baf184  05baf1e0 ; .cxr
05baf188  05baf190 ; .exr
05baf18c  05baf1e0
05baf190  e06d7363
[…]

0:007:x86> .cxr 05baf1e0
eax=05baf4ac ebx=002c1f60 ecx=00000003 edx=00000000 esi=716d9c54 edi=00000010
eip=7757f328 esp=05baf4ac ebp=05baf4fc iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
kernel32!RaiseException+0×58:
7757f328 c9              leave

0:007:x86> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
05baf4fc 7155dead e06d7363 00000001 00000003 kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d 05baf544 715e7954 715e5ac0 Flash10b!DllUnregisterServer+0×1adbe0
05baf550 7150ca25 002c1f7c 4237d4f8 4237c598 Flash10b!DllUnregisterServer+0×1aa2d0
05baf564 7150b5b5 002c1f60 00000010 423d0b28 Flash10b!DllUnregisterServer+0×15c758
05baf588 7150b90a 4237cb38 7150ba0b 00000000 Flash10b!DllUnregisterServer+0×15b2e8
00000000 00000000 00000000 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

0:007:x86> .exr 05baf190
ExceptionAddress: 000000007757f328 (kernel32!RaiseException+0×0000000000000058)
   ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
   Parameter[0]: 0000000019930520
   Parameter[1]: 0000000005baf544
   Parameter[2]: 00000000715e7954

We double check that Flash10b code raised the exception indeed:

0:007:x86> .asm no_code_bytes
Assembly options: no_code_bytes

0:007:x86> ub 7155dead
Flash10b!DllUnregisterServer+0×1adbc4:
7155de91 je      Flash10b!DllUnregisterServer+0×1adbcd (7155de9a)
7155de93 mov     dword ptr [ebp-0Ch],1994000h
7155de9a lea     eax,[ebp-0Ch]
7155de9d push    eax
7155de9e push    dword ptr [ebp-10h]
7155dea1 push    dword ptr [ebp-1Ch]
7155dea4 push    dword ptr [ebp-20h]
7155dea7 call    dword ptr [Flash10b!DllUnregisterServer+0×1ba0a3 (7156a370)] 

0:007:x86> dps 7156a370
7156a370  7757f2cf kernel32!RaiseException
7156a374  77562728 kernel32!WideCharToMultiByte
7156a378  7756c60b kernel32!GlobalFree
7156a37c  77562ddf kernel32!GetCurrentThread
7156a380  775739fe kernel32!SetThreadAffinityMask
7156a384  775654c9 kernel32!VirtualQuery
7156a388  77562e9c kernel32!IsDBCSLeadByte
7156a38c  77561ce5 kernel32!GetACP
7156a390  77565ed6 kernel32!GetCPInfo
7156a394  77561b36 kernel32!MultiByteToWideChar
7156a398  7756eb64 kernel32!ResetEvent
7156a39c  77562c88 kernel32!CreateEventA
7156a3a0  77561a3a kernel32!CloseHandle
7156a3a4  775649c1 kernel32!WaitForMultipleObjects
7156a3a8  77562c64 kernel32!SetEvent
7156a3ac  77560e18 kernel32!InterlockedExchange
7156a3b0  77560e2c kernel32!InterlockedCompareExchange
7156a3b4  77560c79 kernel32!Sleep
7156a3b8  77e935c0 ntdll_77e70000!RtlLeaveCriticalSection
7156a3bc  77e93580 ntdll_77e70000!RtlEnterCriticalSection
7156a3c0  77ea27fe ntdll_77e70000!RtlDeleteCriticalSection
7156a3c4  77ea2512 ntdll_77e70000!RtlInitializeCriticalSection
7156a3c8  77568c28 kernel32!WriteConsoleA
7156a3cc  7757a6dc kernel32!SetStdHandle
7156a3d0  77568c0b kernel32!GetConsoleMode
7156a3d4  77568c75 kernel32!GetConsoleCP
7156a3d8  77564ab7 kernel32!GetStringTypeW
7156a3dc  77576a56 kernel32!GetStringTypeA
7156a3e0  77562d90 kernel32!InitializeCriticalSectionAndSpinCount
7156a3e4  775608d0 kernel32!GetSystemTimeAsFileTime
7156a3e8  77565c3b kernel32!GetEnvironmentStringsW
7156a3ec  77565c13 kernel32!FreeEnvironmentStringsA

I believe the appearance of “dead” in the return address 7155dead was just a coincidence.

So we have found the 3rd-party component that raised e06d7363 C++ exception. Actually Internet search shows that e06d7363 is quite common in many applications.

- Dmitry Vostokov @ DumpAnalysis.org -

If we want to use gflags.exe to enable page heap settings (or any other image file execution options) for a 32-bit executable running on x64 Windows we should use 32-bit version of gflags.exe from 32-bit Debugging Tools for Windows (see windbg.org for quick download links) or if we want to set appropriate registry key manually we should use Wow6432Node branch:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<process name with .exe extension>

Name: GlobalFlag
Type: REG_DWORD
Value: 0x02000000

Name: PageHeapFlags
Type: REG_DWORD
Value: 0x00000003

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we need to dump raw stack data of the given thread to get correct function arguments or to see execution residue. If the process is WOW64 (32-bit running on x64 Windows) and its memory dump was saved using a 64-bit debugger or a process dumper like userdump.exe we have 2 stacks in it:

64-bit stack:

0:007> k
Child-SP          RetAddr           Call Site
00000000`030fee38 00000000`75fcab46 wow64cpu!WaitForMultipleObjects32+0x3a
00000000`030feee0 00000000`75fca14c wow64!RunCpuSimulation+0xa
00000000`030fef10 00000000`77d373db wow64!Wow64LdrpInitialize+0x4b4
00000000`030ff470 00000000`77cf85ce ntdll! ?? ::FNODOBFM::`string'+0x20061
00000000`030ff520 00000000`00000000 ntdll!LdrInitializeThunk+0xe

0:007> r
rax=0000000000000000 rbx=0000000000000002 rcx=0000000000000400
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=0000000075ec374f rsp=00000000030fee38 rbp=0000000005baef38
 r8=000000000000002b  r9=0000000077e9057a r10=0000000000000000
r11=00000000030fee30 r12=000000007efad000 r13=00000000030ffd20
r14=00000000030fee70 r15=0000000075ec3380
iopl=0 nv up ei pl nz ac pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212
wow64cpu!WaitForMultipleObjects32+0×3a:
00000000`75ec374f 418bbda0000000  mov     edi,dword ptr [r13+0A0h] ds:00000000`030ffdc0=00000000

0:007> !teb
Wow64 TEB32 at 000000007efaf000

error InitTypeRead( wow64!_TEB32 )...

Wow64 TEB at 000000007efad000
    ExceptionList:        000000007efaf000
    StackBase:            00000000030ffd20
    StackLimit:           00000000030f8000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efad000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000b44 . 0000000000000b1c
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

32-bit stack:

0:007> .effmach x86
Effective machine: x86 compatible (x86)

0:007:x86> k
ChildEBP RetAddr 
05baee9c 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baef38 775649d9 kernel32!WaitForMultipleObjectsEx+0x11d
05baef54 7761573d kernel32!WaitForMultipleObjects+0x18
05baefc0 77615969 kernel32!WerpReportFaultInternal+0x16d
05baefd4 775ec66f kernel32!WerpReportFault+0x70
05baf060 77eed03e kernel32!UnhandledExceptionFilter+0x1b5
05baf068 77ebf2d0 ntdll_77e70000!__RtlUserThreadStart+0x6f
05baf07c 77f229b3 ntdll_77e70000!_EH4_CallFilterFunc+0x12
05baf0a4 77e93099 ntdll_77e70000!_except_handler4+0x8e
05baf0c8 77e9306b ntdll_77e70000!ExecuteHandler2+0x26
05baf178 77e92eff ntdll_77e70000!ExecuteHandler+0x24
05baf198 7757f328 ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf4fc 7155dead kernel32!RaiseException+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d ComponentA!DllUnregisterServer+0x1adbe0
[...]

0:007:x86> r
eax=00000000 ebx=00000002 ecx=00000003 edx=00000000 esi=00000000 edi=00000000
eip=77e90bc5 esp=05baeea0 ebp=05baef38 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll_77e70000!NtWaitForMultipleObjects+0×15:
77e90bc5 c21400          ret     14h

Unfortunately there is an error when try to get its TEB (TEB32) using !teb command so we can either use dp command to discover stack base and limit:

0:007:x86> !teb
Wow64 TEB32 at 000000007efaf000

error InitTypeRead( wow64!_TEB32 )…

0:007:x86> dp 000000007efaf000 l3
7efaf000  05baef28 05bb0000 05b95000

or use WOW64 extension for this purpose:

0:007:x86> !wow64exts.info

PEB32: 0x7efde000
PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0×7efaf000
TEB64: 0×7efad000

32 bit, StackBase   : 0×5bb0000
        StackLimit  : 0×5b95000
        Deallocation: 0×5ab0000

64 bit, StackBase   : 0x30ffd20
        StackLimit  : 0x30f8000
        Deallocation: 0x30c0000
Wow64 TLS slots:

WOW64_TLS_STACKPTR64:       0x0000000000000000
WOW64_TLS_CPURESERVED:      0x00000000030ffd20
WOW64_TLS_INCPUSIMULATION:  0x0000000000000000
WOW64_TLS_LOCALTHREADHEAP:  0x0000000000000000
WOW64_TLS_EXCEPTIONADDR:    0x000000007757f328
WOW64_TLS_USERCALLBACKDATA: 0x0000000000000000
WOW64_TLS_EXTENDED_FLOAT:   0x0000000000000000
WOW64_TLS_APCLIST:          0x0000000000000000
WOW64_TLS_FILESYSREDIR:     0x0000000000000000
WOW64_TLS_LASTWOWCALL:      0x0000000000000000
WOW64_TLS_WOW64INFO:        0x000000007efde238

In case these methods don’t work and we want to quickly inspect raw memory around the current ESP value we can still use dps esp-xxx esp+xxx command, where xxx is some offset:

0:007:x86> dps esp-10 esp+10
05baee90  00000000
05baee94  00000000
05baee98  00000000
05baee9c  77e90bc5 ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baeea0  7756e91a kernel32!WaitForMultipleObjectsEx+0x11d
05baeea4  00000002
05baeea8  05baeeec
05baeeac  00000001
05baeeb0  00000000

Remember that dp and dps are better to use here than dd, dds or dq, dqs: they automatically take into account the pointer size, 32-bit in x86 mode and 64-bit in native mode:

0:007:x86> .effmach amd64
Effective machine: x64 (AMD64)

0:007> dps esp-10 esp+10
00000000`030fee28  00000000`006d0008
00000000`030fee30  00000000`030ffd20
00000000`030fee38  00000000`7efad000
00000000`030fee40  00000000`00003da8
00000000`030fee48  00000000`75ec3688 wow64cpu!ServiceNoTurbo+0x28

- Dmitry Vostokov @ DumpAnalysis.org -

Consider this redirector file system exception while servicing a page fault:

1: kd> k 100
ChildEBP RetAddr 
afa8b2d4 f53442c4 nt!KeBugCheckEx+0x1b
afa8b2fc f533df48 rdbss!RxExceptionFilter+0xf0
afa8b308 8083a93d rdbss!RxFsdCommonDispatch+0×3ca
afa8b330 8083a498 nt!_except_handler3+0×61
afa8b354 8083a46a nt!ExecuteHandler2+0×26
afa8b404 80816479 nt!ExecuteHandler+0×24
afa8b7bc 808346c4 nt!KiDispatchException+0×131
afa8b824 80834678 nt!CommonDispatchException+0×4a
afa8b848 808401e9 nt!KiExceptionExit+0×186
afa8b8ac f5345d44 nt!IopAllocateIrpPrivate+0xa3
afa8b8f8 f534619c rdbss!RxShadowIoHandler+0xac
afa8b924 f534adb3 rdbss!RxShadowLowIo+0×1a1
afa8b94c f5350fa7 rdbss!RxLowIoSubmit+0×182
afa8b970 f5350dbd rdbss!RxLowIoReadShell+0×7d
afa8ba1c f533d8d9 rdbss!RxCommonRead+0×416
afa8baac f534b9a2 rdbss!RxFsdCommonDispatch+0×320
afa8bacc f52c9a63 rdbss!RxFsdDispatch+0xd3
afa8baec 80840153 mrxsmb!MRxSmbFsdDispatch+0×134
afa8bb00 f71dcc45 nt!IofCallDriver+0×45
afa8bb28 80840153 fltmgr!FltpDispatch+0×6f
afa8bb3c f421576a nt!IofCallDriver+0×45
afa8bb4c f4212621 DriverA!Dispatch+0xa4
afa8bb58 80840153 DriverA!Interface_dispatch+0×53
afa8bb6c f5251aef nt!IofCallDriver+0×45
afa8bb9c f5251bd2 DriverB!PassThrough+0×115
afa8bba8 80840153 DriverB!Dispatch+0×78
afa8bbbc f281ac29 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
afa8bbdc 8082784f DriverC+0×5c29
afa8bc14 80840153 nt!MiResolveMappedFileFault+0×640
afa8bc28 f780f42a nt!IofCallDriver+0×45
afa8bc40 80840153 DriverD+0×42a
afa8bc54 80824b6f nt!IofCallDriver+0×45
afa8bc6c 8082645c nt!IoPageRead+0×109
afa8bcf0 80847650 nt!MiDispatchFault+0xd74
afa8bd4c 80836c2a nt!MmAccessFault+0×9c2
afa8bd4c 77df4749 nt!KiTrap0E+0xdc
0012f568 00000000 0×77df4749

We see that DriverA and DriverB are possibly pass-through and have little influence. For DriverC and DriverD we don’t even have symbol files but they are even further down the stack. Default analysis points to the code that tries to call another driver:

1: kd> !analyze -v
[...]
RDR_FILE_SYSTEM (27)
    If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the exception record and context record. Do a .cxr on the 3rd parameter and then kb to obtain a more informative stack trace. The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined as follows:
     RDBSS_BUG_CHECK_CACHESUP  = 0xca550000,
     RDBSS_BUG_CHECK_CLEANUP   = 0xc1ee0000,
     RDBSS_BUG_CHECK_CLOSE     = 0xc10e0000,
     RDBSS_BUG_CHECK_NTEXCEPT  = 0xbaad0000,
Arguments:
Arg1: baad0080
Arg2: afa8b7d8
Arg3: afa8b4d4
Arg4: 8084014f

EXCEPTION_RECORD:  afa8b7d8 -- (.exr 0xffffffffafa8b7d8)
ExceptionAddress: 8084014f (nt!IofCallDriver+0×00000041)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000044
Attempt to read from address 00000044

CONTEXT:  afa8b4d4 -- (.cxr 0xffffffffafa8b4d4)
eax=00000003 ebx=00000000 ecx=86a62ed8 edx=861e0468 esi=00000000 edi=87083970
eip=8084014f esp=afa8b8a0 ebp=afa8b8ac iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000  efl=00010286
nt!IofCallDriver+0×41:
8084014f ff548638        call    dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

STACK_TEXT: 
afa8b8ac f5345d44 86997a50 87083970 86863604 nt!IofCallDriver+0×41
afa8b8f8 f534619c 87083970 868634e0 e36c7ad8 rdbss!RxShadowIoHandler+0xac
afa8b924 f534adb3 87083970 868634e0 e36c7ad8 rdbss!RxShadowLowIo+0×1a1
afa8b94c f5350fa7 01083970 868634e0 e36c7ad8 rdbss!RxLowIoSubmit+0×182
afa8b970 f5350dbd 87083970 868634e0 e36c7ad8 rdbss!RxLowIoReadShell+0×7d
afa8ba1c f533d8d9 87083970 868634e0 88b56910 rdbss!RxCommonRead+0×416
afa8baac f534b9a2 f5348028 868634e0 88b56910 rdbss!RxFsdCommonDispatch+0×320
afa8bacc f52c9a63 88964768 868634e0 88d6c020 rdbss!RxFsdDispatch+0xd3
afa8baec 80840153 00000000 018634e0 88f372b8 mrxsmb!MRxSmbFsdDispatch+0×134
afa8bb00 f71dcc45 88f372b8 86863628 8843d418 nt!IofCallDriver+0×45
afa8bb28 80840153 88d6c020 868634e0 868634e0 fltmgr!FltpDispatch+0×6f
afa8bb3c f421576a 00000000 88ad73f0 afa8bb6c nt!IofCallDriver+0×45
afa8bb4c f4212621 88d1ea98 868634e0 80840153 DriverA!Dispatch+0xa4
afa8bb58 80840153 88d1ea98 868634e0 88ada7f8 DriverA!Interface_dispatch+0×53
afa8bb6c f5251aef 86863604 88438a78 00000000 nt!IofCallDriver+0×45
afa8bb9c f5251bd2 88ada7f8 868634e0 80840153 DriverB!PassThrough+0×115
afa8bba8 80840153 88ada7f8 868634e0 88b3a0d8 DriverB!Dispatch+0×78
afa8bbbc f281ac29 00000000 87780ca0 afa8bc14 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
afa8bbdc 8082784f 88b3a0d8 81f2afa4 88ab83d8 DriverC+0×5c29
afa8bc14 80840153 88b3a020 868634e0 88433760 nt!MiResolveMappedFileFault+0×640
afa8bc28 f780f42a 8762cf38 f7811f2f 884336a8 nt!IofCallDriver+0×45
afa8bc40 80840153 884336a8 868634e0 868634e0 DriverD+0×42a
afa8bc54 80824b6f 88ab83e8 86e0e438 88ab83d8 nt!IofCallDriver+0×45
afa8bc6c 8082645c 88b56906 88ab8410 88ab83f0 nt!IoPageRead+0×109
afa8bcf0 80847650 00000000 01fc897c c0007f20 nt!MiDispatchFault+0xd74
afa8bd4c 80836c2a 00000000 01fc897c 00000001 nt!MmAccessFault+0×9c2
afa8bd4c 77df4749 00000000 01fc897c 00000001 nt!KiTrap0E+0xdc
0012f568 00000000 00000000 00000000 00000000 0×77df4749

We assume that IofCallDriver has the same parameters as IoCallDriver but they were not passed on the stack. They are passed via ECX and EDX registers:

1: kd> !devobj 86997a50
86997a50: is not a device object

1: kd> !irp 87083970
IRP signature does not match, probably not an IRP

3: kd> .asm no_code_bytes
Assembly options: no_code_bytes

1: kd> ub f5345d44
rdbss!RxShadowIoHandler+0×8d:
f5345d25 mov     dword ptr [ebp-4],ebx
f5345d28 call    dword ptr [rdbss!_imp__IoGetTopLevelIrp (f534711c)]
f5345d2e mov     dword ptr [ebp-20h],eax
f5345d31 push    ebx
f5345d32 call    dword ptr [rdbss!_imp__IoSetTopLevelIrp (f5347120)]
f5345d38 mov     edx,dword ptr [ebp-24h]
f5345d3b mov     ecx,dword ptr [ebp-28h]
f5345d3e call    dword ptr [rdbss!_imp_IofCallDriver (f5347204)]

We have the value of EBP at the time of the crash at IofCallDriver+0×41 so we need to see how it had changed:

1: kd> uf nt!IofCallDriver
Flow analysis was incomplete, some code may be missing
nt!IofCallDriver:
8084011b mov     edi,edi
8084011d push    ebp
8084011e mov     ebp,esp
80840120 mov     eax,dword ptr [nt!pIofCallDriver (808a5000)]
80840125 test    eax,eax
80840127 jne     nt!IofCallDriver+0xe (80859d6b)

nt!IofCallDriver+0x15:
8084012d dec     byte ptr [edx+23h]
80840130 cmp     byte ptr [edx+23h],0
80840134 jle     nt!IofCallDriver+0x1e (80859d72)

nt!IofCallDriver+0x2c:
8084013a mov     eax,dword ptr [edx+60h]
8084013d sub     eax,24h
80840140 push    esi
80840141 mov     dword ptr [edx+60h],eax
80840144 mov     dword ptr [eax+14h],ecx
80840147 movzx   eax,byte ptr [eax]
8084014a mov     esi,dword ptr [ecx+8]
8084014d push    edx
8084014e push    ecx
8084014f call    dword ptr [esi+eax*4+38h]
80840153 pop     esi
80840154 pop     ebp
80840155 ret

nt!IofCallDriver+0xe:
80859d6b push    dword ptr [ebp+4]
80859d6e call    eax
80859d70 pop     ebp
80859d71 ret

nt!IofCallDriver+0x1e:
80859d72 push    0
80859d74 push    0
80859d76 push    0
80859d78 push    edx
80859d79 push    35h
80859d7b call    nt!KeBugCheckEx (8087c485)
80859d80 int     3
80859d81 pop     ebp
80859d82 jmp     eax

We see the standard prolog and therefore EBP points to the old EBP from RxShadowIoHandler:

1: kd> dp ebp l1
afa8b8ac  afa8b8f8

We have the value of the pointer to a device object, which is invalid:

1: kd> dp afa8b8f8-28 l1
afa8b8d0  86a62ed8

1: kd> !devobj 86a62ed8
86a62ed8: is not a device object

and the value of the pointer to an IRP which is valid:

1: kd> dp afa8b8f8-24 l1
afa8b8d4  861e0468

1: kd> !irp 861e0468
Irp is active with 4 stacks 4 is current (= 0x861e0544)
 Mdl=88ab8410: No System Buffer: Thread 8846e020:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
>[  3, 0]   0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel
       86a62ed8: is not a device object
 rdbss!RxShadowIrpCompletion
   Args: 00008000 00000000 00008000 00000000

From the full disassembly code of IofCallDriver we see that ECX and EDX values had not been changed so we could get our parameters from them too.

Device object is invalid in IRP too and it points to a NULL driver object:

1: kd> dt _DEVICE_OBJECT 86a62ed8
nt!_DEVICE_OBJECT
   +0×000 Type             : 0
   +0×002 Size             : 0
   +0×004 ReferenceCount   : 0
   +0×008 DriverObject     : (null)
   +0×00c NextDevice       : (null)
   +0×010 AttachedDevice   : (null)
   +0×014 CurrentIrp       : (null)
   +0×018 Timer            : (null)
   +0×01c Flags            : 0
   +0×020 Characteristics  : 0
   +0×024 Vpb              : 0×00040000 _VPB
   +0×028 DeviceExtension  : (null)
   +0×02c DeviceType       : 0×86a62f04
   +0×030 StackSize        : 4 ”
   +0×034 Queue            : __unnamed
   +0×05c AlignmentRequirement : 0
   +0×060 DeviceQueue      : _KDEVICE_QUEUE
   +0×074 Dpc              : _KDPC
   +0×094 ActiveThreadCount : 0
   +0×098 SecurityDescriptor : (null)
   +0×09c DeviceLock       : _KEVENT
   +0×0ac SectorSize       : 0
   +0×0ae Spare1           : 0
   +0×0b0 DeviceObjectExtension : 0×86a62f88 _DEVOBJ_EXTENSION
   +0×0b4 Reserved         : 0×86a62f88

IofCallDriver tried to call a function pointer No.3 from the driver major dispatch table:

1: kd> dt _DRIVER_OBJECT
nt!_DRIVER_OBJECT
   +0x000 Type             : Int2B
   +0x002 Size             : Int2B
   +0x004 DeviceObject     : Ptr32 _DEVICE_OBJECT
   +0x008 Flags            : Uint4B
   +0x00c DriverStart      : Ptr32 Void
   +0x010 DriverSize       : Uint4B
   +0x014 DriverSection    : Ptr32 Void
   +0x018 DriverExtension  : Ptr32 _DRIVER_EXTENSION
   +0x01c DriverName       : _UNICODE_STRING
   +0x024 HardwareDatabase : Ptr32 _UNICODE_STRING
   +0x028 FastIoDispatch   : Ptr32 _FAST_IO_DISPATCH
   +0x02c DriverInit       : Ptr32     long
   +0x030 DriverStartIo    : Ptr32     void
   +0x034 DriverUnload     : Ptr32     void
   +0×038 MajorFunction    : [28] Ptr32     long

The resulted effective address is a NULL code pointer (EAX=3, major code and ESI is NULL):

call    dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

Now we come back to our IRP to examine a file object:

     cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel

1: kd> !fileobj 86ff7028

\Userdata\[…]\Application Suite\ApplicationSuite.RUS

Device Object: 0×87300aa8   \Driver\DriverE
Vpb: 0×8843b280
Event signalled
Access: Read SharedRead SharedDelete

Flags:  0x18c0040
 Cache Supported
 Handle Created
 Fast IO Read
 Remote Origin

FsContext: 0xe453fda0 FsContext2: 0xe26904b8
CurrentByteOffset: 0
Cache Data:
  Section Object Pointers: 86b80a9c
  Shared Cache Map: 00000000

Here we see another DriverE and associated drivers from its device stack:

1: kd> !devstack 0×87300aa8
  !DevObj   !DrvObj            !DevExt   ObjectName
  865f9cc8  \Driver\DriverF   865f9d80 
  8737aaf0  \Driver\DriverG    8737aba8 
  87155280  \Driver\DriverH    87155338 
> 87300aa8  \Driver\DriverE       87300b60  BlockVolume1
!DevNode 88b6eac0 :
  DeviceInst is “STORAGE\Volume\{[…]}”
  ServiceName is “volsnap”

- Dmitry Vostokov @ DumpAnalysis.org -

Finally I reused component-like frame previously belonged to an MCSD certificate displayed at Programming Research office where I worked 6-7 years ago (books in a row there are all about C++):

Old Frame

Today I dug it from dusty corners of my apartment and inserted a page 1152 snapshot as promised:

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump

A snapshot of memory. Can be sequential (cell by cell) or selective.

References: Memory Dump - A Mathematical Definition

Synonyms: crash dump, core dump.

Antonyms:

Also: user dump, kernel dump, complete dump, full dump, minidump.

- Dmitry Vostokov @ DumpAnalysis.org -

This small case study continues where Not using checklists common mistake case study left, after identifying the blocked LPC thread in ServiceA process. We know that ServiceA always asks (coupled with) ServiceB and, indeed, when looking at the latter stack trace collection we see a GUI thread showing a dialog box:

0:000> ~13kc

ntdll!KiFastSystemCallRet
USER32!NtUserWaitMessage
USER32!InternalDialogBox
USER32!DialogBoxIndirectParamAorW
USER32!DialogBoxIndirectParamW
comdlg32!NewGetFileName
comdlg32!NewGetSaveFileName
comdlg32!GetFileName
comdlg32!GetSaveFileNameW
WARNING: Stack unwind information not available. Following frames may be wrong.
DllA!DllEntryPoint
ServiceB!Initialize
[…]
ServiceB!ServiceThread
kernel32!BaseThreadStart

From function names we can infer that the thread was displaying a “Save File As” common dialog box but the service was not allowed to interact with a desktop. This made the service blocked and, in return, made ServiceA blocked too. The suggestion was to enable ServiceB to interact with the desktop and keep an eye on DllA.

- Dmitry Vostokov @ DumpAnalysis.org -