Software Diagnostics Library

If we want to use gflags.exe to enable page heap settings (or any other image file execution options) for a 32-bit executable running on x64 Windows we should use 32-bit version of gflags.exe from 32-bit Debugging Tools for Windows (see windbg.org for quick download links) or if we want to set appropriate registry key manually we should use Wow6432Node branch:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<process name with .exe extension>

Name: GlobalFlag
Type: REG_DWORD
Value: 0x02000000

Name: PageHeapFlags
Type: REG_DWORD
Value: 0x00000003

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we need to dump raw stack data of the given thread to get correct function arguments or to see execution residue. If the process is WOW64 (32-bit running on x64 Windows) and its memory dump was saved using a 64-bit debugger or a process dumper like userdump.exe we have 2 stacks in it:

64-bit stack:

0:007> k
Child-SP          RetAddr           Call Site
00000000`030fee38 00000000`75fcab46 wow64cpu!WaitForMultipleObjects32+0x3a
00000000`030feee0 00000000`75fca14c wow64!RunCpuSimulation+0xa
00000000`030fef10 00000000`77d373db wow64!Wow64LdrpInitialize+0x4b4
00000000`030ff470 00000000`77cf85ce ntdll! ?? ::FNODOBFM::`string'+0x20061
00000000`030ff520 00000000`00000000 ntdll!LdrInitializeThunk+0xe

0:007> r
rax=0000000000000000 rbx=0000000000000002 rcx=0000000000000400
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=0000000075ec374f rsp=00000000030fee38 rbp=0000000005baef38
 r8=000000000000002b  r9=0000000077e9057a r10=0000000000000000
r11=00000000030fee30 r12=000000007efad000 r13=00000000030ffd20
r14=00000000030fee70 r15=0000000075ec3380
iopl=0 nv up ei pl nz ac pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212
wow64cpu!WaitForMultipleObjects32+0×3a:
00000000`75ec374f 418bbda0000000  mov     edi,dword ptr [r13+0A0h] ds:00000000`030ffdc0=00000000

0:007> !teb
Wow64 TEB32 at 000000007efaf000

error InitTypeRead( wow64!_TEB32 )...

Wow64 TEB at 000000007efad000
    ExceptionList:        000000007efaf000
    StackBase:            00000000030ffd20
    StackLimit:           00000000030f8000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efad000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000b44 . 0000000000000b1c
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

32-bit stack:

0:007> .effmach x86
Effective machine: x86 compatible (x86)

0:007:x86> k
ChildEBP RetAddr 
05baee9c 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baef38 775649d9 kernel32!WaitForMultipleObjectsEx+0x11d
05baef54 7761573d kernel32!WaitForMultipleObjects+0x18
05baefc0 77615969 kernel32!WerpReportFaultInternal+0x16d
05baefd4 775ec66f kernel32!WerpReportFault+0x70
05baf060 77eed03e kernel32!UnhandledExceptionFilter+0x1b5
05baf068 77ebf2d0 ntdll_77e70000!__RtlUserThreadStart+0x6f
05baf07c 77f229b3 ntdll_77e70000!_EH4_CallFilterFunc+0x12
05baf0a4 77e93099 ntdll_77e70000!_except_handler4+0x8e
05baf0c8 77e9306b ntdll_77e70000!ExecuteHandler2+0x26
05baf178 77e92eff ntdll_77e70000!ExecuteHandler+0x24
05baf198 7757f328 ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf4fc 7155dead kernel32!RaiseException+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d ComponentA!DllUnregisterServer+0x1adbe0
[...]

0:007:x86> r
eax=00000000 ebx=00000002 ecx=00000003 edx=00000000 esi=00000000 edi=00000000
eip=77e90bc5 esp=05baeea0 ebp=05baef38 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll_77e70000!NtWaitForMultipleObjects+0×15:
77e90bc5 c21400          ret     14h

Unfortunately there is an error when try to get its TEB (TEB32) using !teb command so we can either use dp command to discover stack base and limit:

0:007:x86> !teb
Wow64 TEB32 at 000000007efaf000

error InitTypeRead( wow64!_TEB32 )…

0:007:x86> dp 000000007efaf000 l3
7efaf000  05baef28 05bb0000 05b95000

or use WOW64 extension for this purpose:

0:007:x86> !wow64exts.info

PEB32: 0x7efde000
PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0×7efaf000
TEB64: 0×7efad000

32 bit, StackBase   : 0×5bb0000
        StackLimit  : 0×5b95000
        Deallocation: 0×5ab0000

64 bit, StackBase   : 0x30ffd20
        StackLimit  : 0x30f8000
        Deallocation: 0x30c0000
Wow64 TLS slots:

WOW64_TLS_STACKPTR64:       0x0000000000000000
WOW64_TLS_CPURESERVED:      0x00000000030ffd20
WOW64_TLS_INCPUSIMULATION:  0x0000000000000000
WOW64_TLS_LOCALTHREADHEAP:  0x0000000000000000
WOW64_TLS_EXCEPTIONADDR:    0x000000007757f328
WOW64_TLS_USERCALLBACKDATA: 0x0000000000000000
WOW64_TLS_EXTENDED_FLOAT:   0x0000000000000000
WOW64_TLS_APCLIST:          0x0000000000000000
WOW64_TLS_FILESYSREDIR:     0x0000000000000000
WOW64_TLS_LASTWOWCALL:      0x0000000000000000
WOW64_TLS_WOW64INFO:        0x000000007efde238

In case these methods don’t work and we want to quickly inspect raw memory around the current ESP value we can still use dps esp-xxx esp+xxx command, where xxx is some offset:

0:007:x86> dps esp-10 esp+10
05baee90  00000000
05baee94  00000000
05baee98  00000000
05baee9c  77e90bc5 ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baeea0  7756e91a kernel32!WaitForMultipleObjectsEx+0x11d
05baeea4  00000002
05baeea8  05baeeec
05baeeac  00000001
05baeeb0  00000000

Remember that dp and dps are better to use here than dd, dds or dq, dqs: they automatically take into account the pointer size, 32-bit in x86 mode and 64-bit in native mode:

0:007:x86> .effmach amd64
Effective machine: x64 (AMD64)

0:007> dps esp-10 esp+10
00000000`030fee28  00000000`006d0008
00000000`030fee30  00000000`030ffd20
00000000`030fee38  00000000`7efad000
00000000`030fee40  00000000`00003da8
00000000`030fee48  00000000`75ec3688 wow64cpu!ServiceNoTurbo+0x28

- Dmitry Vostokov @ DumpAnalysis.org -

Consider this redirector file system exception while servicing a page fault:

1: kd> k 100
ChildEBP RetAddr 
afa8b2d4 f53442c4 nt!KeBugCheckEx+0x1b
afa8b2fc f533df48 rdbss!RxExceptionFilter+0xf0
afa8b308 8083a93d rdbss!RxFsdCommonDispatch+0×3ca
afa8b330 8083a498 nt!_except_handler3+0×61
afa8b354 8083a46a nt!ExecuteHandler2+0×26
afa8b404 80816479 nt!ExecuteHandler+0×24
afa8b7bc 808346c4 nt!KiDispatchException+0×131
afa8b824 80834678 nt!CommonDispatchException+0×4a
afa8b848 808401e9 nt!KiExceptionExit+0×186
afa8b8ac f5345d44 nt!IopAllocateIrpPrivate+0xa3
afa8b8f8 f534619c rdbss!RxShadowIoHandler+0xac
afa8b924 f534adb3 rdbss!RxShadowLowIo+0×1a1
afa8b94c f5350fa7 rdbss!RxLowIoSubmit+0×182
afa8b970 f5350dbd rdbss!RxLowIoReadShell+0×7d
afa8ba1c f533d8d9 rdbss!RxCommonRead+0×416
afa8baac f534b9a2 rdbss!RxFsdCommonDispatch+0×320
afa8bacc f52c9a63 rdbss!RxFsdDispatch+0xd3
afa8baec 80840153 mrxsmb!MRxSmbFsdDispatch+0×134
afa8bb00 f71dcc45 nt!IofCallDriver+0×45
afa8bb28 80840153 fltmgr!FltpDispatch+0×6f
afa8bb3c f421576a nt!IofCallDriver+0×45
afa8bb4c f4212621 DriverA!Dispatch+0xa4
afa8bb58 80840153 DriverA!Interface_dispatch+0×53
afa8bb6c f5251aef nt!IofCallDriver+0×45
afa8bb9c f5251bd2 DriverB!PassThrough+0×115
afa8bba8 80840153 DriverB!Dispatch+0×78
afa8bbbc f281ac29 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
afa8bbdc 8082784f DriverC+0×5c29
afa8bc14 80840153 nt!MiResolveMappedFileFault+0×640
afa8bc28 f780f42a nt!IofCallDriver+0×45
afa8bc40 80840153 DriverD+0×42a
afa8bc54 80824b6f nt!IofCallDriver+0×45
afa8bc6c 8082645c nt!IoPageRead+0×109
afa8bcf0 80847650 nt!MiDispatchFault+0xd74
afa8bd4c 80836c2a nt!MmAccessFault+0×9c2
afa8bd4c 77df4749 nt!KiTrap0E+0xdc
0012f568 00000000 0×77df4749

We see that DriverA and DriverB are possibly pass-through and have little influence. For DriverC and DriverD we don’t even have symbol files but they are even further down the stack. Default analysis points to the code that tries to call another driver:

1: kd> !analyze -v
[...]
RDR_FILE_SYSTEM (27)
    If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the exception record and context record. Do a .cxr on the 3rd parameter and then kb to obtain a more informative stack trace. The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined as follows:
     RDBSS_BUG_CHECK_CACHESUP  = 0xca550000,
     RDBSS_BUG_CHECK_CLEANUP   = 0xc1ee0000,
     RDBSS_BUG_CHECK_CLOSE     = 0xc10e0000,
     RDBSS_BUG_CHECK_NTEXCEPT  = 0xbaad0000,
Arguments:
Arg1: baad0080
Arg2: afa8b7d8
Arg3: afa8b4d4
Arg4: 8084014f

EXCEPTION_RECORD:  afa8b7d8 -- (.exr 0xffffffffafa8b7d8)
ExceptionAddress: 8084014f (nt!IofCallDriver+0×00000041)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000044
Attempt to read from address 00000044

CONTEXT:  afa8b4d4 -- (.cxr 0xffffffffafa8b4d4)
eax=00000003 ebx=00000000 ecx=86a62ed8 edx=861e0468 esi=00000000 edi=87083970
eip=8084014f esp=afa8b8a0 ebp=afa8b8ac iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000  efl=00010286
nt!IofCallDriver+0×41:
8084014f ff548638        call    dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

STACK_TEXT: 
afa8b8ac f5345d44 86997a50 87083970 86863604 nt!IofCallDriver+0×41
afa8b8f8 f534619c 87083970 868634e0 e36c7ad8 rdbss!RxShadowIoHandler+0xac
afa8b924 f534adb3 87083970 868634e0 e36c7ad8 rdbss!RxShadowLowIo+0×1a1
afa8b94c f5350fa7 01083970 868634e0 e36c7ad8 rdbss!RxLowIoSubmit+0×182
afa8b970 f5350dbd 87083970 868634e0 e36c7ad8 rdbss!RxLowIoReadShell+0×7d
afa8ba1c f533d8d9 87083970 868634e0 88b56910 rdbss!RxCommonRead+0×416
afa8baac f534b9a2 f5348028 868634e0 88b56910 rdbss!RxFsdCommonDispatch+0×320
afa8bacc f52c9a63 88964768 868634e0 88d6c020 rdbss!RxFsdDispatch+0xd3
afa8baec 80840153 00000000 018634e0 88f372b8 mrxsmb!MRxSmbFsdDispatch+0×134
afa8bb00 f71dcc45 88f372b8 86863628 8843d418 nt!IofCallDriver+0×45
afa8bb28 80840153 88d6c020 868634e0 868634e0 fltmgr!FltpDispatch+0×6f
afa8bb3c f421576a 00000000 88ad73f0 afa8bb6c nt!IofCallDriver+0×45
afa8bb4c f4212621 88d1ea98 868634e0 80840153 DriverA!Dispatch+0xa4
afa8bb58 80840153 88d1ea98 868634e0 88ada7f8 DriverA!Interface_dispatch+0×53
afa8bb6c f5251aef 86863604 88438a78 00000000 nt!IofCallDriver+0×45
afa8bb9c f5251bd2 88ada7f8 868634e0 80840153 DriverB!PassThrough+0×115
afa8bba8 80840153 88ada7f8 868634e0 88b3a0d8 DriverB!Dispatch+0×78
afa8bbbc f281ac29 00000000 87780ca0 afa8bc14 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
afa8bbdc 8082784f 88b3a0d8 81f2afa4 88ab83d8 DriverC+0×5c29
afa8bc14 80840153 88b3a020 868634e0 88433760 nt!MiResolveMappedFileFault+0×640
afa8bc28 f780f42a 8762cf38 f7811f2f 884336a8 nt!IofCallDriver+0×45
afa8bc40 80840153 884336a8 868634e0 868634e0 DriverD+0×42a
afa8bc54 80824b6f 88ab83e8 86e0e438 88ab83d8 nt!IofCallDriver+0×45
afa8bc6c 8082645c 88b56906 88ab8410 88ab83f0 nt!IoPageRead+0×109
afa8bcf0 80847650 00000000 01fc897c c0007f20 nt!MiDispatchFault+0xd74
afa8bd4c 80836c2a 00000000 01fc897c 00000001 nt!MmAccessFault+0×9c2
afa8bd4c 77df4749 00000000 01fc897c 00000001 nt!KiTrap0E+0xdc
0012f568 00000000 00000000 00000000 00000000 0×77df4749

We assume that IofCallDriver has the same parameters as IoCallDriver but they were not passed on the stack. They are passed via ECX and EDX registers:

1: kd> !devobj 86997a50
86997a50: is not a device object

1: kd> !irp 87083970
IRP signature does not match, probably not an IRP

3: kd> .asm no_code_bytes
Assembly options: no_code_bytes

1: kd> ub f5345d44
rdbss!RxShadowIoHandler+0×8d:
f5345d25 mov     dword ptr [ebp-4],ebx
f5345d28 call    dword ptr [rdbss!_imp__IoGetTopLevelIrp (f534711c)]
f5345d2e mov     dword ptr [ebp-20h],eax
f5345d31 push    ebx
f5345d32 call    dword ptr [rdbss!_imp__IoSetTopLevelIrp (f5347120)]
f5345d38 mov     edx,dword ptr [ebp-24h]
f5345d3b mov     ecx,dword ptr [ebp-28h]
f5345d3e call    dword ptr [rdbss!_imp_IofCallDriver (f5347204)]

We have the value of EBP at the time of the crash at IofCallDriver+0×41 so we need to see how it had changed:

1: kd> uf nt!IofCallDriver
Flow analysis was incomplete, some code may be missing
nt!IofCallDriver:
8084011b mov     edi,edi
8084011d push    ebp
8084011e mov     ebp,esp
80840120 mov     eax,dword ptr [nt!pIofCallDriver (808a5000)]
80840125 test    eax,eax
80840127 jne     nt!IofCallDriver+0xe (80859d6b)

nt!IofCallDriver+0x15:
8084012d dec     byte ptr [edx+23h]
80840130 cmp     byte ptr [edx+23h],0
80840134 jle     nt!IofCallDriver+0x1e (80859d72)

nt!IofCallDriver+0x2c:
8084013a mov     eax,dword ptr [edx+60h]
8084013d sub     eax,24h
80840140 push    esi
80840141 mov     dword ptr [edx+60h],eax
80840144 mov     dword ptr [eax+14h],ecx
80840147 movzx   eax,byte ptr [eax]
8084014a mov     esi,dword ptr [ecx+8]
8084014d push    edx
8084014e push    ecx
8084014f call    dword ptr [esi+eax*4+38h]
80840153 pop     esi
80840154 pop     ebp
80840155 ret

nt!IofCallDriver+0xe:
80859d6b push    dword ptr [ebp+4]
80859d6e call    eax
80859d70 pop     ebp
80859d71 ret

nt!IofCallDriver+0x1e:
80859d72 push    0
80859d74 push    0
80859d76 push    0
80859d78 push    edx
80859d79 push    35h
80859d7b call    nt!KeBugCheckEx (8087c485)
80859d80 int     3
80859d81 pop     ebp
80859d82 jmp     eax

We see the standard prolog and therefore EBP points to the old EBP from RxShadowIoHandler:

1: kd> dp ebp l1
afa8b8ac  afa8b8f8

We have the value of the pointer to a device object, which is invalid:

1: kd> dp afa8b8f8-28 l1
afa8b8d0  86a62ed8

1: kd> !devobj 86a62ed8
86a62ed8: is not a device object

and the value of the pointer to an IRP which is valid:

1: kd> dp afa8b8f8-24 l1
afa8b8d4  861e0468

1: kd> !irp 861e0468
Irp is active with 4 stacks 4 is current (= 0x861e0544)
 Mdl=88ab8410: No System Buffer: Thread 8846e020:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
>[  3, 0]   0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel
       86a62ed8: is not a device object
 rdbss!RxShadowIrpCompletion
   Args: 00008000 00000000 00008000 00000000

From the full disassembly code of IofCallDriver we see that ECX and EDX values had not been changed so we could get our parameters from them too.

Device object is invalid in IRP too and it points to a NULL driver object:

1: kd> dt _DEVICE_OBJECT 86a62ed8
nt!_DEVICE_OBJECT
   +0×000 Type             : 0
   +0×002 Size             : 0
   +0×004 ReferenceCount   : 0
   +0×008 DriverObject     : (null)
   +0×00c NextDevice       : (null)
   +0×010 AttachedDevice   : (null)
   +0×014 CurrentIrp       : (null)
   +0×018 Timer            : (null)
   +0×01c Flags            : 0
   +0×020 Characteristics  : 0
   +0×024 Vpb              : 0×00040000 _VPB
   +0×028 DeviceExtension  : (null)
   +0×02c DeviceType       : 0×86a62f04
   +0×030 StackSize        : 4 ”
   +0×034 Queue            : __unnamed
   +0×05c AlignmentRequirement : 0
   +0×060 DeviceQueue      : _KDEVICE_QUEUE
   +0×074 Dpc              : _KDPC
   +0×094 ActiveThreadCount : 0
   +0×098 SecurityDescriptor : (null)
   +0×09c DeviceLock       : _KEVENT
   +0×0ac SectorSize       : 0
   +0×0ae Spare1           : 0
   +0×0b0 DeviceObjectExtension : 0×86a62f88 _DEVOBJ_EXTENSION
   +0×0b4 Reserved         : 0×86a62f88

IofCallDriver tried to call a function pointer No.3 from the driver major dispatch table:

1: kd> dt _DRIVER_OBJECT
nt!_DRIVER_OBJECT
   +0x000 Type             : Int2B
   +0x002 Size             : Int2B
   +0x004 DeviceObject     : Ptr32 _DEVICE_OBJECT
   +0x008 Flags            : Uint4B
   +0x00c DriverStart      : Ptr32 Void
   +0x010 DriverSize       : Uint4B
   +0x014 DriverSection    : Ptr32 Void
   +0x018 DriverExtension  : Ptr32 _DRIVER_EXTENSION
   +0x01c DriverName       : _UNICODE_STRING
   +0x024 HardwareDatabase : Ptr32 _UNICODE_STRING
   +0x028 FastIoDispatch   : Ptr32 _FAST_IO_DISPATCH
   +0x02c DriverInit       : Ptr32     long
   +0x030 DriverStartIo    : Ptr32     void
   +0x034 DriverUnload     : Ptr32     void
   +0×038 MajorFunction    : [28] Ptr32     long

The resulted effective address is a NULL code pointer (EAX=3, major code and ESI is NULL):

call    dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

Now we come back to our IRP to examine a file object:

     cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel

1: kd> !fileobj 86ff7028

\Userdata\[…]\Application Suite\ApplicationSuite.RUS

Device Object: 0×87300aa8   \Driver\DriverE
Vpb: 0×8843b280
Event signalled
Access: Read SharedRead SharedDelete

Flags:  0x18c0040
 Cache Supported
 Handle Created
 Fast IO Read
 Remote Origin

FsContext: 0xe453fda0 FsContext2: 0xe26904b8
CurrentByteOffset: 0
Cache Data:
  Section Object Pointers: 86b80a9c
  Shared Cache Map: 00000000

Here we see another DriverE and associated drivers from its device stack:

1: kd> !devstack 0×87300aa8
  !DevObj   !DrvObj            !DevExt   ObjectName
  865f9cc8  \Driver\DriverF   865f9d80 
  8737aaf0  \Driver\DriverG    8737aba8 
  87155280  \Driver\DriverH    87155338 
> 87300aa8  \Driver\DriverE       87300b60  BlockVolume1
!DevNode 88b6eac0 :
  DeviceInst is “STORAGE\Volume\{[…]}”
  ServiceName is “volsnap”

- Dmitry Vostokov @ DumpAnalysis.org -

Finally I adopted the following process for book reviews:

1. Start reading a book.

2. After sometime put an initial reveiw on this blog.

3. Iteratively and incrementally revise it during read. For example, add additional bits as soon as I discover something good or bad worth mentioning in the book review.

4. When the book is finished put its review blog post into a category: From Cover To Cover.

5. Put the review on Amazon and mark this appropriately by putting its blog post into another category: Reviewed on Amazon.

- Dmitry Vostokov @ LiterateScientist.com -

- Dmitry Vostokov @ Memory Religion Portal -

Finally I reused component-like frame previously belonged to an MCSD certificate displayed at Programming Research office where I worked 6-7 years ago (books in a row there are all about C++):

Old Frame

Today I dug it from dusty corners of my apartment and inserted a page 1152 snapshot as promised:

- Dmitry Vostokov @ DumpAnalysis.org -

- Dmitry Vostokov @ Memory Religion Portal -

Memory Dump

A snapshot of memory. Can be sequential (cell by cell) or selective.

References: Memory Dump - A Mathematical Definition

Synonyms: crash dump, core dump.

Antonyms:

Also: user dump, kernel dump, complete dump, full dump, minidump.

- Dmitry Vostokov @ DumpAnalysis.org -

This small case study continues where Not using checklists common mistake case study left, after identifying the blocked LPC thread in ServiceA process. We know that ServiceA always asks (coupled with) ServiceB and, indeed, when looking at the latter stack trace collection we see a GUI thread showing a dialog box:

0:000> ~13kc

ntdll!KiFastSystemCallRet
USER32!NtUserWaitMessage
USER32!InternalDialogBox
USER32!DialogBoxIndirectParamAorW
USER32!DialogBoxIndirectParamW
comdlg32!NewGetFileName
comdlg32!NewGetSaveFileName
comdlg32!GetFileName
comdlg32!GetSaveFileNameW
WARNING: Stack unwind information not available. Following frames may be wrong.
DllA!DllEntryPoint
ServiceB!Initialize
[…]
ServiceB!ServiceThread
kernel32!BaseThreadStart

From function names we can infer that the thread was displaying a “Save File As” common dialog box but the service was not allowed to interact with a desktop. This made the service blocked and, in return, made ServiceA blocked too. The suggestion was to enable ServiceB to interact with the desktop and keep an eye on DllA.

- Dmitry Vostokov @ DumpAnalysis.org -

Got it in the post yesterday shipped via courier from Amazon:

Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (PRO-Developer)

I originally expected it to be a paperback book but as a publisher myself I welcome the final MS Press decision to make it a hardcover. It is more solid. I plan to read it from cover to cover again, starting from tomorrow. I would like to thank authors for including the reference to www.dumpanalysis.org in the conclusion part of the last chapter 14 dedicated to crash dump analysis. This came as a nice surprise for me and I’m going to install a special plaque in the office and put its picture later on my blog.

- Dmitry Vostokov @ DumpAnalysis.org -

Supportability is similar to serviceability and while working on DebugWare book I realized that writing support tools needs its own buzz word like model-driven software design. Hence SDSD acronym was born a few days ago:

SDSD

Supportability-Driven Software Design

or

Support-Driven Software Design

or

Serviceability-Driven Software Design

Thinking about where to insert requirements, architecture and design led me to another acronym:

RADII

Requirements, Architecture, Design, Implementation and Improvement

The plural form of radius signifies the fact that there is a plurality of ways how SDLC can be implemented. Improvement is similar to Maintenance.

- Dmitry Vostokov @ DumpAnalysis.org -

I read this book from cover to cover while flying on a plane from Dublin to St. Petersburg and back. That was so wonderful reading experience - I couldn’t put the book down during those flights. I recall that I visited the Department of Mathematics a few times when I studied Chemistry in Moscow State University although at that time I knew next to nothing about Russian mathematicians. The book touched me so deeply that I bought the main work of Florensky: The Pillar and Ground of the Truth, the history of Russian philosophy and several books explaining Orthodox Church. This is the best mathematics history book I have ever read, my feelings perhaps comparable to those that I experienced when I finished reading Mathematics: The Loss of Certainty by Morris Kline but that was more than 20 years ago.

Naming Infinity: A True Story of Religious Mysticism and Mathematical Creativity

- Dmitry Vostokov @ LiterateScientist.com -

One of the common mistakes that I observe is to habitually stick to certain WinDbg commands to recognize patterns. One example is !locks command used to find out any wait chains and deadlock conditions among threads. Recently a service process was reported to be hang and !locks command showed no blocked threads:

0:000> !locks
CritSec +18caf94 at 018CAF94
LockCount          -2
RecursionCount     1
OwningThread       58e8
EntryCount         0
ContentionCount    0
*** Locked

CritSec +18cc7c4 at 018CC7C4
LockCount          -2
RecursionCount     1
OwningThread       58e8
EntryCount         0
ContentionCount    0
*** Locked

The number of threads waiting for the lock is 0 (this calculation is explained in the MSDN article): 

0:000> ? ((-1) - (-2)) >> 2
Evaluate expression: 0 = 00000000

In the past, for that hang sevice memory dumps, !locks command always showed LockCount values corresponding to several waiting threads. Therefore, an engineer assumed that the dump was taken at some random time, not at the time the service was hanging, and asked for a new right dump. The mistake here is that the engineer didn’t look at the corresponding thread stack trace that shows the characteristic pattern of the blocked thread waiting for a reply from an LRPC call:

0:000> ~~[58e8]kc 100

ntdll!KiFastSystemCallRet
ntdll!NtRequestWaitReplyPort
RPCRT4!LRPC_CCALL::SendReceive
RPCRT4!I_RpcSendReceive
RPCRT4!NdrSendReceive
RPCRT4!NdrClientCall2
ServiceA!foo
[…]
ServiceA!bar
RPCRT4!NdrStubCall2
RPCRT4!NdrServerCall2
RPCRT4!DispatchToStubInCNoAvrf
RPCRT4!RPC_INTERFACE::DispatchToStubWorker
RPCRT4!RPC_INTERFACE::DispatchToStub
RPCRT4!RPC_INTERFACE::DispatchToStubWithObject
RPCRT4!LRPC_SCALL::DealWithRequestMessage
RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest
RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls
RPCRT4!RecvLotsaCallsWrapper
RPCRT4!BaseCachedThreadRoutine
RPCRT4!ThreadStartRoutine
kernel32!BaseThreadStart

We don’t see other blocked threads and wait chains because the dump was saved as soon as the freezing condition was detected: the service didn’t allow a user connection to proceed. If more users tried to connect we would have seen critical section wait chains that are absent in this dump.

To prevent such mistakes checklists are indispensable. For one example, see Crash Dump Analysis Checklist. You can also order it in print:

WinDbg: A Reference Poster and Learning Cards

- Dmitry Vostokov @ DumpAnalysis.org -

A picture taken during my recent visit to Peterhof (one of the 7 wonders of Russia):

- Dmitry Vostokov @ DumpAnalysis.org -