Archive for the ‘Assembly Language’ Category

On Self Dumps of Secure String API

Tuesday, July 14th, 2009

Sometimes we get crashes with the following stack trace pattern:

0:000> kv 100
ChildEBP RetAddr  Args to Child             
0012cecc 7c91df4a 7c8094fe 00000002 0012cef8 ntdll!KiFastSystemCallRet
0012ced0 7c8094fe 00000002 0012cef8 00000001 ntdll!ZwWaitForMultipleObjects+0xc
0012cf6c 7c80a085 00000002 0012d09c 00000000 kernel32!WaitForMultipleObjectsEx+0x12c
0012cf88 6990763c 00000002 0012d09c 00000000 kernel32!WaitForMultipleObjects+0x18
0012d91c 699082b1 0012f090 00000001 00198312 faultrep!StartDWException+0×5df
0012e990 7c8635d1 0012f090 00000001 00000000 faultrep!ReportFault+0×533
0012f030 78138a09 0012f090 00000022 c000000d kernel32!UnhandledExceptionFilter+0×587
0012f368 781443d1 00000000 00000000 00000000 msvcr80!_invoke_watson+0xc4

0012f38c 0040b02f 0012f538 00000104 004f80a0 msvcr80!strcat_s+0×29
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f39c 0012f538 00000104 00430848 0012f538 Application+0xb020

0:000> .exptr 0012f090

----- Exception record at 0012f040:
ExceptionAddress: 781443d1 (msvcr80!strcat_s+0x00000029)
   ExceptionCode: c000000d
  ExceptionFlags: 00000000
NumberParameters: 0

----- Context record at 0012f098:
eax=f2c4dacf ebx=00000000 ecx=00000002 edx=7c91e514 esi=00000022 edi=00000000
eip=781443d1 esp=0012f36c ebp=004f7658 iopl=0 nv up ei ng nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000286
msvcr80!strcat_s+0x29:
781443d1 83c414          add     esp,14h

0:000> !error c000000d
Error code: (NTSTATUS) 0xc000000d (3221225485) - An invalid parameter was passed to a service or function.

We clearly see that the crash involved strcat_s but we don’t see any invalid instruction or access violation call. The ADD instruction is perfectly valid above. However, if we disassemble EIP backwards we would see the call to _invalid_parameter function and it looks like it has a name association with Dr. Watson:

0:000> ub 781443d1
msvcr80!strcat_s+0×1c:
781443c4 5e              pop     esi
781443c5 8930            mov     dword ptr [eax],esi
781443c7 53              push    ebx
781443c8 53              push    ebx
781443c9 53              push    ebx
781443ca 53              push    ebx
781443cb 53              push    ebx
781443cc e89f46ffff      call    msvcr80!_invalid_parameter (78138a70)

0:000> uf 78138a70
Flow analysis was incomplete, some code may be missing
msvcr80!_invoke_watson:
78138945 55              push    ebp
78138946 8dac2458fdffff  lea     ebp,[esp-2A8h]
7813894d 81ec28030000    sub     esp,328h

[...]

msvcr80!_invalid_parameter:
78138a70 55              push    ebp
78138a71 8bec            mov     ebp,esp
78138a73 ff3528401c78    push    dword ptr [msvcr80!__pInvalidArgHandler (781c4028)]
78138a79 e85ba1ffff      call    msvcr80!_decode_pointer (78132bd9)
78138a7e 85c0            test    eax,eax
78138a80 59              pop     ecx
78138a81 7403            je      msvcr80!_invalid_parameter+0×16 (78138a86)

msvcr80!_invalid_parameter+0x13:
78138a83 5d              pop     ebp
78138a84 ffe0            jmp     eax

msvcr80!_invalid_parameter+0x16:
78138a86 6a02            push    2
78138a88 e806330000      call    msvcr80!_crt_debugger_hook (7813bd93)
78138a8d 59              pop     ecx
78138a8e 5d              pop     ebp
78138a8f e9b1feffff      jmp     msvcr80!_invoke_watson (78138945)

According to MSDN documentation, _s secure functions by default use a postmortem debugger mechanism:

Secure-enhanced CRT parameter validation

So we have something that is similar to Self-Dump pattern here. The same parameter checking is seen in the case of C++ STL exceptions. In case of custom unhandled exception filters not resorting to WER faulrep.dll other stacks can show process termination, for example, with wcscpy_s:

0:000> kL 100
ChildEBP RetAddr 
0111cb64 7c947c39 ntdll!KiFastSystemCallRet
0111cb68 7c80202b ntdll!ZwTerminateProcess+0xc
0111cb78 78138a2b kernel32!TerminateProcess+0×20
0111ceb4 78144ba1 MSVCR80!_invoke_watson+0xe6

0111ced8 67dbb47d MSVCR80!wcscpy_s+0×29
0111cf00 67dbc93b Application!CopyName+0×5d
[…]

Here the specified size of the destination buffer was smaller than the size of source NULL-terminated strings, which was a good thing anyway: old strcpy or strcat function would definitely caused buffer overflow effects. Now we have a nice side effect too, the dump is saved, ready for a postmortem analysis and subsequent code improvement.

- Dmitry Vostokov @ DumpAnalysis.org -

The Intelligent Memory Movement

Tuesday, July 14th, 2009

IMM, The Intelligent Memory Movement borrows the following Intel meta-opcode for its slogan:

NoIndirection: MOV OPERAND, IMM ; Memory first, Being second

Please assemble and join me in the direction of this movement!

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 87)

Friday, July 10th, 2009

Sometimes we get rare or hardware-related bugchecks and these might have been influenced by hardware virtualization (that is also a software and could have its own defects). Therefore it is beneficial to recognize when a system is running under a VM (Virtualized System pattern):

Memory Dumps from Xen-virtualized Windows

Memory dumps from VMware images (Virtual PC diagnostics in post comments)

Memory Dumps from Hyper-Virtualized Windows

For example, we get the following bugcheck and stack trace for the first processor:

0: kd> !analyze -v
[...]
CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 00000060, Clock interrupt time out interval in nominal clock ticks.
Arg2: 00000000, 0.
Arg3: 805d8120, The PRCB address of the hung processor.
Arg4: 00000001, 0.

CURRENT_IRQL:  1c

STACK_TEXT: 
8d283694 816df8a5 nt!KeBugCheckEx+0x1e
8d2836c8 816e163d nt!KeUpdateRunTime+0xd5
8d2836c8 84617008 nt!KeUpdateSystemTime+0xed
WARNING: Frame IP not in any known module. Following frames may be wrong.
8d283748 816f46a1 0×84617008
8d283758 816fa6aa nt!HvlpInitiateHypercall+0×21
8d283784 8166aca5 nt!HvlNotifyLongSpinWait+0×2b
8d2837a0 816cce7e nt!KeFlushSingleTb+0xc4
8d283808 81681db4 nt!MmAccessFault+0xc1d
8d283808 816dd033 nt!KiTrap0E+0xdc

8d28389c 8168ed58 nt!memcpy+0×33
8d283954 816712bf nt!MmCopyToCachedPage+0×1193
8d2839ec 81663053 nt!CcMapAndCopy+0×210
8d283a74 8c688218 nt!CcFastCopyWrite+0×283
8d283b98 8c40badc Ntfs!NtfsCopyWriteA+0×23e
8d283bcc 8c40bcab fltmgr!FltpPerformFastIoCall+0×22e
8d283bf8 8c41dc30 fltmgr!FltpPassThroughFastIo+0×7d
8d283c3c 818471cd fltmgr!FltpFastIoWrite+0×146
8d283d38 8167ec7a nt!NtWriteFile+0×34c
8d283d38 77115e74 nt!KiFastCallEntry+0×12a
01cfee80 00000000 0×77115e74

The thread was servicing a page fault. Notice the gap between KeUpdateSystemTime and HvlpInitiateHypercall. This is normal and consistent code if we look closer:

0: kd> .asm no_code_bytes
Assembly options: no_code_bytes

0: kd> uf HvlpInitiateHypercall
nt!HvlpInitiateHypercall:
816f4680 push    edi
816f4681 push    esi
816f4682 push    ebx
816f4683 mov     eax,dword ptr [esp+10h]
816f4687 mov     edx,dword ptr [esp+14h]
816f468b mov     ecx,dword ptr [esp+18h]
816f468f mov     ebx,dword ptr [esp+1Ch]
816f4693 mov     esi,dword ptr [esp+20h]
816f4697 mov     edi,dword ptr [esp+24h]
816f469b call    dword ptr [nt!HvlpHypercallCodeVa (8176bb8c)]
816f46a1 pop     ebx
816f46a2 pop     esi
816f46a3 pop     edi
816f46a4 ret     18h

0: kd> dp 8176bb8c l1
8176bb8c  84617000

0: kd> uf 84617000
84617000 or      eax,80000000h
84617005 vmcall
84617008ret

We have the address of RET instruction (84617008) on the stack trace:

0: kd> kv
ChildEBP RetAddr  Args to Child             
8d283694 816df8a5 00000101 00000060 00000000 nt!KeBugCheckEx+0x1e
8d2836c8 816e163d 84e1521b 000000d1 8d283784 nt!KeUpdateRunTime+0xd5
8d2836c8 8461700884e1521b 000000d1 8d283784 nt!KeUpdateSystemTime+0xed (FPO: [0,2] TrapFrame @ 8d2836d8)
WARNING: Frame IP not in any known module. Following frames may be wrong.
8d283748 816f46a1 84e6c900 22728000 8172e28c 0×84617008
8d283758 816fa6aa 00010008 00000000 22728000 nt!HvlpInitiateHypercall+0×21 (FPO: [6,3,0])
8d283784 8166aca5 22728000 00000000 00000000 nt!HvlNotifyLongSpinWait+0×2b
[…]

The second processor is busy too:

0: kd> !running

System Processors 3 (affinity mask)
  Idle Processors 0

Prcbs  Current   Next   
  0    8172c920  84e6c900            ................
  1    805d8120  85138030  85a50d78  …………….

0: kd> !thread 85138030
THREAD 85138030  Cid 0564.11c8  Teb: 7ff9f000 Win32Thread: 00000000 RUNNING on processor 1
IRP List:
    85ab5d00: (0006,01fc) Flags: 00000884  Mdl: 00000000
    85445ab8: (0006,0094) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 98a7d558
Owning Process            84f0d938       Image:         Application.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      695643         Ticks: 224 (0:00:00:03.500)
Context Switch Count      20            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x705e2679
Stack Init a1d13000 Current a1d10a70 Base a1d13000 Limit a1d10000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr  Args to Child             
00000000 00000000 00000000 00000000 00000000 0×0

Because we have (Limit, Current, Base) triple for our thread we check its execution residue on kernel raw stack. We find traces of a hypercall too:

0: kd> dds a1d10000 a1d13000
[...]
a1d1215c  816fa6da nt!HvlEndSystemInterrupt+0x20
a1d12160  40000070
a1d12164  00000000
a1d12168  81608838 hal!HalEndSystemInterrupt+0x7a
a1d1216c  805d8000
a1d12170  a1d12180
a1d12174  81618cc9 hal!HalpIpiHandler+0x189
a1d12178  84f4841b
a1d1217c  000000e1
a1d12180  a1d1222c
a1d12184  84617008
a1d12188  badb0d00
a1d1218c  00000000
a1d12190  81665699 nt!RtlWalkFrameChain+0x58
a1d12194  816659c4 nt!RtlWalkFrameChain+0x377
a1d12198  11c4649e
a1d1219c  00000002
a1d121a0  00000003
a1d121a4  85478444
a1d121a8  0000001f
a1d121ac  00000000
a1d121b0  00000000
a1d121b4  816f46a1 nt!HvlpInitiateHypercall+0×21
a1d121b8  8172c800 nt!KiInitialPCR
a1d121bc  85138030
a1d121c0  85a72ac0
a1d121c4  8171b437 nt!HvlSwitchVirtualAddressSpace+0×28
a1d121c8  00010001
a1d121cc  00000000
a1d121d0  85a72ac0
a1d121d4  85138030
a1d121d8  8172c802 nt!KiInitialPCR+0×2
a1d121dc  00000000
a1d121e0  85138030
a1d121e4  816fa6da nt!HvlEndSystemInterrupt+0×20
a1d121e8  40000070
a1d121ec  00000000
a1d121f0  81608838 hal!HalEndSystemInterrupt+0×7a
a1d121f4  816f46a1 nt!HvlpInitiateHypercall+0×21
a1d121f8  805d8000
a1d121fc  85138030
a1d12200  805dc1e0
a1d12204  8171b437 nt!HvlSwitchVirtualAddressSpace+0×28
a1d12208  00010001
a1d1220c  00000000
a1d12210  00000000
a1d12214  81608468 hal!HalpDispatchSoftwareInterrupt+0×5e
a1d12218  00000000
a1d1221c  00000000
a1d12220  00000206
a1d12224  a1d12240
a1d12228  81608668 hal!HalpCheckForSoftwareInterrupt+0×64
a1d1222c  00000002
a1d12230  00000000
a1d12234  816086a8 hal!KfLowerIrql
a1d12238  00000000
a1d1223c  00000002
a1d12240  a1d12250
a1d12244  8160870c hal!KfLowerIrql+0×64
a1d12248  00000000
a1d1224c  00000000
a1d12250  a1d12294
a1d12254  816e035a nt!KiSwapThread+0×477
a1d12258  85138030
a1d1225c  851380b8
a1d12260  805d8120
a1d12264  0014d1f8
[…]

Looking at raw stack further we can even see that it was processing a page fault too and manually reconstruct its stack trace:

[...]
a1d1074c  85aef510
a1d10750  a1d10768
a1d10754  81678976 nt!IofCallDriver+0×63
a1d10758  84c87d50
a1d1075c  85aef510
a1d10760  00000000
a1d10764  84c87d50
a1d10768  a1d10784
a1d1076c  8166d74e nt!IoPageRead+0×172
a1d10770  85138030
a1d10774  84a1352c
a1d10778  84a134f8
a1d1077c  84a13538
a1d10780  84c87d50
a1d10784  a1d10840
a1d10788  816abf07 nt!MiDispatchFault+0xd14
a1d1078c  00000043
a1d10790  85138030
a1d10794  84a13538
a1d10798  84a1350c
a1d1079c  84a1352c
a1d107a0  8174c800 nt!MmSystemCacheWs
a1d107a4  00000000
a1d107a8  85138030
a1d107ac  a5397bf8
a1d107b0  85b01c48
a1d107b4  00000000
a1d107b8  00000000
a1d107bc  a5397bf8
a1d107c0  84a1358c
a1d107c4  a1d10864
a1d107c8  00000000
a1d107cc  8463a590
a1d107d0  84a134f8
a1d107d4  c0518000
a1d107d8  00000000
a1d107dc  00000000
a1d107e0  00000028
a1d107e4  a1d107f4
a1d107e8  00000000
a1d107ec  00000038
a1d107f0  859f5640
a1d107f4  a4bfa390
a1d107f8  00000000
a1d107fc  00000000
a1d10800  00000000
a1d10804  a1d10938
a1d10808  818652bd nt!MmCreateSection+0×98f
a1d1080c  00000000
a1d10810  846652e8
a1d10814  00000000
a1d10818  00000000
a1d1081c  00000000
a1d10820  00000028
a1d10824  00000000
a1d10828  00000080
a1d1082c  0000000a
a1d10830  85ae1c98
a1d10834  85ae1c20
a1d10838  00000000
a1d1083c  00000000
a1d10840  a1d108b8
a1d10844  816cd325 nt!MmAccessFault+0×10c6
a1d10848  a3000000
a1d1084c  a5397bf8
a1d10850  00000000
a1d10854  8174c800 nt!MmSystemCacheWs
a1d10858  00000000
a1d1085c  00000000
a1d10860  a5397bf8
a1d10864  00000000
[…]

0: kd> k L=a1d10750 a1d10750 a1d10750
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
a1d10750 81678976 0xa1d10750
a1d10768 8166d74e nt!IofCallDriver+0x63
a1d10784 816abf07 nt!IoPageRead+0x172
a1d10840 816cd325 nt!MiDispatchFault+0xd14
a1d108b8 816f0957 nt!MmAccessFault+0x10c6
a1d10924 8181c952 nt!MmCheckCachedPageState+0x801
a1d109b0 8c60f850 nt!CcCopyRead+0x435
a1d109dc 8c613c52 Ntfs!NtfsCachedRead+0x13b
a1d10abc 8c612b6f Ntfs!NtfsCommonRead+0x105a
a1d10b2c 81678976 Ntfs!NtfsFsdRead+0x273
a1d10b44 8c40cba7 nt!IofCallDriver+0x63
a1d10b68 8c40d7c7 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x251
a1d10ba0 8c40dbe7 fltmgr!FltPerformSynchronousIo+0xb9
a1d10c10 9ca680e5 fltmgr!FltReadFile+0x2ed
[...]

A bit of reverse engineering offtopic here: I think that the absence of a trap means that cache processing NTFS code reuses page fault handling code.

- Dmitry Vostokov @ DumpAnalysis.org -

Hunting for a Driver

Monday, July 6th, 2009

Consider this redirector file system exception while servicing a page fault:

1: kd> k 100
ChildEBP RetAddr 
afa8b2d4 f53442c4 nt!KeBugCheckEx+0x1b
afa8b2fc f533df48 rdbss!RxExceptionFilter+0xf0
afa8b308 8083a93d rdbss!RxFsdCommonDispatch+0×3ca
afa8b330 8083a498 nt!_except_handler3+0×61
afa8b354 8083a46a nt!ExecuteHandler2+0×26
afa8b404 80816479 nt!ExecuteHandler+0×24
afa8b7bc 808346c4 nt!KiDispatchException+0×131
afa8b824 80834678 nt!CommonDispatchException+0×4a

afa8b848 808401e9 nt!KiExceptionExit+0×186
afa8b8ac f5345d44 nt!IopAllocateIrpPrivate+0xa3
afa8b8f8 f534619c rdbss!RxShadowIoHandler+0xac
afa8b924 f534adb3 rdbss!RxShadowLowIo+0×1a1
afa8b94c f5350fa7 rdbss!RxLowIoSubmit+0×182
afa8b970 f5350dbd rdbss!RxLowIoReadShell+0×7d
afa8ba1c f533d8d9 rdbss!RxCommonRead+0×416
afa8baac f534b9a2 rdbss!RxFsdCommonDispatch+0×320
afa8bacc f52c9a63 rdbss!RxFsdDispatch+0xd3
afa8baec 80840153 mrxsmb!MRxSmbFsdDispatch+0×134
afa8bb00 f71dcc45 nt!IofCallDriver+0×45
afa8bb28 80840153 fltmgr!FltpDispatch+0×6f
afa8bb3c f421576a nt!IofCallDriver+0×45
afa8bb4c f4212621 DriverA!Dispatch+0xa4
afa8bb58 80840153 DriverA!Interface_dispatch+0×53

afa8bb6c f5251aef nt!IofCallDriver+0×45
afa8bb9c f5251bd2 DriverB!PassThrough+0×115
afa8bba8 80840153 DriverB!Dispatch+0×78

afa8bbbc f281ac29 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
afa8bbdc 8082784f DriverC+0×5c29
afa8bc14 80840153 nt!MiResolveMappedFileFault+0×640
afa8bc28 f780f42a nt!IofCallDriver+0×45
afa8bc40 80840153 DriverD+0×42a
afa8bc54 80824b6f nt!IofCallDriver+0×45
afa8bc6c 8082645c nt!IoPageRead+0×109
afa8bcf0 80847650 nt!MiDispatchFault+0xd74
afa8bd4c 80836c2a nt!MmAccessFault+0×9c2

afa8bd4c 77df4749 nt!KiTrap0E+0xdc
0012f568 00000000 0×77df4749

We see that DriverA and DriverB are possibly pass-through and have little influence. For DriverC and DriverD we don’t even have symbol files but they are even further down the stack. Default analysis points to the code that tries to call another driver:

1: kd> !analyze -v
[...]
RDR_FILE_SYSTEM (27)
    If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the exception record and context record. Do a .cxr on the 3rd parameter and then kb to obtain a more informative stack trace. The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined as follows:
     RDBSS_BUG_CHECK_CACHESUP  = 0xca550000,
     RDBSS_BUG_CHECK_CLEANUP   = 0xc1ee0000,
     RDBSS_BUG_CHECK_CLOSE     = 0xc10e0000,
     RDBSS_BUG_CHECK_NTEXCEPT  = 0xbaad0000,
Arguments:
Arg1: baad0080
Arg2: afa8b7d8
Arg3: afa8b4d4
Arg4: 8084014f

EXCEPTION_RECORD:  afa8b7d8 -- (.exr 0xffffffffafa8b7d8)
ExceptionAddress: 8084014f (nt!IofCallDriver+0×00000041)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000044
Attempt to read from address 00000044

CONTEXT:  afa8b4d4 -- (.cxr 0xffffffffafa8b4d4)
eax=00000003 ebx=00000000 ecx=86a62ed8 edx=861e0468 esi=00000000 edi=87083970
eip=8084014f esp=afa8b8a0 ebp=afa8b8ac iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000  efl=00010286
nt!IofCallDriver+0×41:
8084014f ff548638        call    dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

STACK_TEXT: 
afa8b8ac f5345d44 86997a50 87083970 86863604 nt!IofCallDriver+0×41
afa8b8f8 f534619c 87083970 868634e0 e36c7ad8 rdbss!RxShadowIoHandler+0xac
afa8b924 f534adb3 87083970 868634e0 e36c7ad8 rdbss!RxShadowLowIo+0×1a1
afa8b94c f5350fa7 01083970 868634e0 e36c7ad8 rdbss!RxLowIoSubmit+0×182
afa8b970 f5350dbd 87083970 868634e0 e36c7ad8 rdbss!RxLowIoReadShell+0×7d
afa8ba1c f533d8d9 87083970 868634e0 88b56910 rdbss!RxCommonRead+0×416
afa8baac f534b9a2 f5348028 868634e0 88b56910 rdbss!RxFsdCommonDispatch+0×320
afa8bacc f52c9a63 88964768 868634e0 88d6c020 rdbss!RxFsdDispatch+0xd3
afa8baec 80840153 00000000 018634e0 88f372b8 mrxsmb!MRxSmbFsdDispatch+0×134
afa8bb00 f71dcc45 88f372b8 86863628 8843d418 nt!IofCallDriver+0×45
afa8bb28 80840153 88d6c020 868634e0 868634e0 fltmgr!FltpDispatch+0×6f
afa8bb3c f421576a 00000000 88ad73f0 afa8bb6c nt!IofCallDriver+0×45
afa8bb4c f4212621 88d1ea98 868634e0 80840153 DriverA!Dispatch+0xa4
afa8bb58 80840153 88d1ea98 868634e0 88ada7f8 DriverA!Interface_dispatch+0×53
afa8bb6c f5251aef 86863604 88438a78 00000000 nt!IofCallDriver+0×45
afa8bb9c f5251bd2 88ada7f8 868634e0 80840153 DriverB!PassThrough+0×115
afa8bba8 80840153 88ada7f8 868634e0 88b3a0d8 DriverB!Dispatch+0×78
afa8bbbc f281ac29 00000000 87780ca0 afa8bc14 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
afa8bbdc 8082784f 88b3a0d8 81f2afa4 88ab83d8 DriverC+0×5c29
afa8bc14 80840153 88b3a020 868634e0 88433760 nt!MiResolveMappedFileFault+0×640
afa8bc28 f780f42a 8762cf38 f7811f2f 884336a8 nt!IofCallDriver+0×45
afa8bc40 80840153 884336a8 868634e0 868634e0 DriverD+0×42a
afa8bc54 80824b6f 88ab83e8 86e0e438 88ab83d8 nt!IofCallDriver+0×45
afa8bc6c 8082645c 88b56906 88ab8410 88ab83f0 nt!IoPageRead+0×109
afa8bcf0 80847650 00000000 01fc897c c0007f20 nt!MiDispatchFault+0xd74
afa8bd4c 80836c2a 00000000 01fc897c 00000001 nt!MmAccessFault+0×9c2
afa8bd4c 77df4749 00000000 01fc897c 00000001 nt!KiTrap0E+0xdc
0012f568 00000000 00000000 00000000 00000000 0×77df4749

We assume that IofCallDriver has the same parameters as IoCallDriver but they were not passed on the stack. They are passed via ECX and EDX registers:

1: kd> !devobj 86997a50
86997a50: is not a device object

1: kd> !irp 87083970
IRP signature does not match, probably not an IRP

3: kd> .asm no_code_bytes
Assembly options: no_code_bytes

1: kd> ub f5345d44
rdbss!RxShadowIoHandler+0×8d:
f5345d25 mov     dword ptr [ebp-4],ebx
f5345d28 call    dword ptr [rdbss!_imp__IoGetTopLevelIrp (f534711c)]
f5345d2e mov     dword ptr [ebp-20h],eax
f5345d31 push    ebx
f5345d32 call    dword ptr [rdbss!_imp__IoSetTopLevelIrp (f5347120)]
f5345d38 mov     edx,dword ptr [ebp-24h]
f5345d3b mov     ecx,dword ptr [ebp-28h]

f5345d3e call    dword ptr [rdbss!_imp_IofCallDriver (f5347204)]

We have the value of EBP at the time of the crash at IofCallDriver+0×41 so we need to see how it had changed:

1: kd> uf nt!IofCallDriver
Flow analysis was incomplete, some code may be missing
nt!IofCallDriver:
8084011b mov     edi,edi
8084011d push    ebp
8084011e mov     ebp,esp
80840120 mov     eax,dword ptr [nt!pIofCallDriver (808a5000)]
80840125 test    eax,eax
80840127 jne     nt!IofCallDriver+0xe (80859d6b)

nt!IofCallDriver+0x15:
8084012d dec     byte ptr [edx+23h]
80840130 cmp     byte ptr [edx+23h],0
80840134 jle     nt!IofCallDriver+0x1e (80859d72)

nt!IofCallDriver+0x2c:
8084013a mov     eax,dword ptr [edx+60h]
8084013d sub     eax,24h
80840140 push    esi
80840141 mov     dword ptr [edx+60h],eax
80840144 mov     dword ptr [eax+14h],ecx
80840147 movzx   eax,byte ptr [eax]
8084014a mov     esi,dword ptr [ecx+8]
8084014d push    edx
8084014e push    ecx
8084014f call    dword ptr [esi+eax*4+38h]
80840153 pop     esi
80840154 pop     ebp
80840155 ret

nt!IofCallDriver+0xe:
80859d6b push    dword ptr [ebp+4]
80859d6e call    eax
80859d70 pop     ebp
80859d71 ret

nt!IofCallDriver+0x1e:
80859d72 push    0
80859d74 push    0
80859d76 push    0
80859d78 push    edx
80859d79 push    35h
80859d7b call    nt!KeBugCheckEx (8087c485)
80859d80 int     3
80859d81 pop     ebp
80859d82 jmp     eax

We see the standard prolog and therefore EBP points to the old EBP from RxShadowIoHandler:

1: kd> dp ebp l1
afa8b8ac  afa8b8f8

We have the value of the pointer to a device object, which is invalid:

1: kd> dp afa8b8f8-28 l1
afa8b8d0  86a62ed8

1: kd> !devobj 86a62ed8
86a62ed8: is not a device object

and the value of the pointer to an IRP which is valid:

1: kd> dp afa8b8f8-24 l1
afa8b8d4  861e0468

1: kd> !irp 861e0468
Irp is active with 4 stacks 4 is current (= 0x861e0544)
 Mdl=88ab8410: No System Buffer: Thread 8846e020:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

   Args: 00000000 00000000 00000000 00000000
>[  3, 0]   0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel
       86a62ed8: is not a device object
 rdbss!RxShadowIrpCompletion
   Args: 00008000 00000000 00008000 00000000

From the full disassembly code of IofCallDriver we see that ECX and EDX values had not been changed so we could get our parameters from them too.

Device object is invalid in IRP too and it points to a NULL driver object:

1: kd> dt _DEVICE_OBJECT 86a62ed8
nt!_DEVICE_OBJECT
   +0×000 Type             : 0
   +0×002 Size             : 0
   +0×004 ReferenceCount   : 0
   +0×008 DriverObject     : (null)
   +0×00c NextDevice       : (null)
   +0×010 AttachedDevice   : (null)
   +0×014 CurrentIrp       : (null)
   +0×018 Timer            : (null)
   +0×01c Flags            : 0
   +0×020 Characteristics  : 0
   +0×024 Vpb              : 0×00040000 _VPB
   +0×028 DeviceExtension  : (null)
   +0×02c DeviceType       : 0×86a62f04
   +0×030 StackSize        : 4 ”
   +0×034 Queue            : __unnamed
   +0×05c AlignmentRequirement : 0
   +0×060 DeviceQueue      : _KDEVICE_QUEUE
   +0×074 Dpc              : _KDPC
   +0×094 ActiveThreadCount : 0
   +0×098 SecurityDescriptor : (null)
   +0×09c DeviceLock       : _KEVENT
   +0×0ac SectorSize       : 0
   +0×0ae Spare1           : 0
   +0×0b0 DeviceObjectExtension : 0×86a62f88 _DEVOBJ_EXTENSION
   +0×0b4 Reserved         : 0×86a62f88

IofCallDriver tried to call a function pointer No.3 from the driver major dispatch table:

1: kd> dt _DRIVER_OBJECT
nt!_DRIVER_OBJECT
   +0x000 Type             : Int2B
   +0x002 Size             : Int2B
   +0x004 DeviceObject     : Ptr32 _DEVICE_OBJECT
   +0x008 Flags            : Uint4B
   +0x00c DriverStart      : Ptr32 Void
   +0x010 DriverSize       : Uint4B
   +0x014 DriverSection    : Ptr32 Void
   +0x018 DriverExtension  : Ptr32 _DRIVER_EXTENSION
   +0x01c DriverName       : _UNICODE_STRING
   +0x024 HardwareDatabase : Ptr32 _UNICODE_STRING
   +0x028 FastIoDispatch   : Ptr32 _FAST_IO_DISPATCH
   +0x02c DriverInit       : Ptr32     long
   +0x030 DriverStartIo    : Ptr32     void
   +0x034 DriverUnload     : Ptr32     void
   +0×038 MajorFunction    : [28] Ptr32     long

The resulted effective address is a NULL code pointer (EAX=3, major code and ESI is NULL):

call    dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

Now we come back to our IRP to examine a file object:

     cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel

1: kd> !fileobj 86ff7028

\Userdata\[…]\Application Suite\ApplicationSuite.RUS

Device Object: 0×87300aa8   \Driver\DriverE
Vpb: 0×8843b280
Event signalled
Access: Read SharedRead SharedDelete

Flags:  0x18c0040
 Cache Supported
 Handle Created
 Fast IO Read
 Remote Origin

FsContext: 0xe453fda0 FsContext2: 0xe26904b8
CurrentByteOffset: 0
Cache Data:
  Section Object Pointers: 86b80a9c
  Shared Cache Map: 00000000

Here we see another DriverE and associated drivers from its device stack:

1: kd> !devstack 0×87300aa8
  !DevObj   !DrvObj            !DevExt   ObjectName
  865f9cc8  \Driver\DriverF   865f9d80 
  8737aaf0  \Driver\DriverG    8737aba8 
  87155280  \Driver\DriverH    87155338 
> 87300aa8  \Driver\DriverE       87300b60  BlockVolume1
!DevNode 88b6eac0 :
  DeviceInst is “STORAGE\Volume\{[…]}”
  ServiceName is “volsnap”

- Dmitry Vostokov @ DumpAnalysis.org -

Comprehensive Rootkit Book

Monday, May 25th, 2009

Found today this book while browsing Amazon:

The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System

Buy from Amazon

Intrigued, I searched for its table of contents and found the author’s site:

Book TOC

Looks enough comprehensive so I pre-ordered the book and plan to write a review later from windows internals and memory dump analysis perspective.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 84)

Friday, May 15th, 2009

Sometimes the assembly code looks almost wild (not like generated by your favourite compiler). For example (that also shows .NET runtime native unhandled exception processing):

0:000> kL 100
ChildEBP RetAddr 
0014dbb4 77189254 ntdll!KiFastSystemCallRet
0014dbb8 75fec244 ntdll!ZwWaitForSingleObject+0xc
0014dc28 75fec1b2 kernel32!WaitForSingleObjectEx+0xbe
0014dc3c 72605389 kernel32!WaitForSingleObject+0x12
0014dc6c 726058e7 mscorwks!ClrWaitForSingleObject+0x24
0014e128 72608084 mscorwks!RunWatson+0x1df
0014e86c 7260874a mscorwks!DoFaultReportWorker+0xb59
0014e8a8 72657452 mscorwks!DoFaultReport+0xc3
0014e8cc 7265c0c7 mscorwks!WatsonLastChance+0x3f
0014e924 7265c173 mscorwks!CLRAddVectoredHandlers+0x209
0014e92c 7603f4be mscorwks!InternalUnhandledExceptionFilter+0x22
0014e9e8 771a85b7 kernel32!UnhandledExceptionFilter+0×127
0014e9f0 77139a14 ntdll!__RtlUserThreadStart+0×6f
0014ea04 771340f4 ntdll!_EH4_CallFilterFunc+0×12
0014ea2c 77189b99 ntdll!_except_handler4+0×8e
0014ea50 77189b6b ntdll!ExecuteHandler2+0×26
0014eb00 771899f7 ntdll!ExecuteHandler+0×24
0014eb00 03ca0141 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0014ee28 634c2f42 0×3ca0141
0014ee34 67715e44 System_ni+0×132f42
0014ee70 72431b4c System_ServiceProcess_ni+0×25e44
0014ee80 724421f9 mscorwks!CallDescrWorker+0×33
0014ef00 72456571 mscorwks!CallDescrWorkerWithHandler+0xa3
0014f03c 724565a4 mscorwks!MethodDesc::CallDescr+0×19c
0014f058 724565c2 mscorwks!MethodDesc::CallTargetWorker+0×1f
0014f070 724afac5 mscorwks!MethodDescCallSite::CallWithValueTypes+0×1a
0014f1d4 724af9e5 mscorwks!ClassLoader::RunMain+0×223
0014f43c 724aff35 mscorwks!Assembly::ExecuteMainMethod+0xa6
0014f90c 724b011f mscorwks!SystemDomain::ExecuteMainMethod+0×456
0014f95c 724b004f mscorwks!ExecuteEXE+0×59
0014f9a4 72f57c24 mscorwks!_CorExeMain+0×15c
0014f9b4 75fe4911 mscoree!_CorExeMain+0×2c
0014f9c0 7716e4b6 kernel32!BaseThreadInitThunk+0xe
0014fa00 7716e489 ntdll!__RtlUserThreadStart+0×23
0014fa18 00000000 ntdll!_RtlUserThreadStart+0×1b

We set exception context:

0:000> kv 100
ChildEBP RetAddr  Args to Child             
[...]
0014e9e8 771a85b7 0014ea18 77139a14 00000000 kernel32!UnhandledExceptionFilter+0×127 (FPO: [SEH])
[…]

0:000> .exptr 0014ea18

----- Exception record at 0014eb18:
ExceptionAddress: 03ca0141
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000000
Attempt to read from address 00000000

----- Context record at 0014eb34:
eax=00000001 ebx=08394ff8 ecx=00000000 edx=00000001 esi=056a2a94 edi=00000000
eip=03ca0141 esp=0014ee00 ebp=0014ee28 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246
03ca0141 3909            cmp     dword ptr [ecx],ecx  ds:0023:00000000=????????

Then we disassemble the code at crash point and it looks strange including calls through DS data segment:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> u 03ca0141
03ca0141 cmp     dword ptr [ecx],ecx
03ca0143 call    dword ptr ds:[36067C0h]
03ca0149 mov     ecx,dword ptr [esi+5Ch]
03ca014c cmp     dword ptr [ecx],ecx
03ca014e call    dword ptr ds:[3606D10h]
03ca0154 mov     dword ptr [ebp-1Ch],0
03ca015b mov     dword ptr [ebp-18h],0FCh
03ca0162 push    3CA0180h

However further disassembly finally reaches RET instruction:

0:000> u
03ca0167 jmp     03ca0169
03ca0169 movzx   edx,byte ptr [ebp-24h]
03ca016d mov     ecx,dword ptr [ebp-28h]
03ca0170 call    System_ServiceProcess_ni+0x25140 (67715140)
03ca0175 pop     eax
03ca0176 jmp     eax
03ca0178 lea     esp,[ebp-0Ch]
03ca017b pop     ebx

0:000> u
03ca017c pop     esi
03ca017d pop     edi
03ca017e pop     ebp
03ca017f ret

03ca0180 mov     dword ptr [ebp-18h],0
03ca0187 jmp     03ca0178
03ca0189 add     byte ptr [eax],al
03ca018b add     byte ptr [eax],al

and backward disassembling shows the matching function prolog code:

0:000> ub 03ca0141
03ca0127 movzx   eax,byte ptr [ebp-24h]
03ca012b test    eax,eax
03ca012d je      03ca0154
03ca012f cmp     dword ptr [esi+60h],0
03ca0133 je      03ca013e
03ca0135 mov     ecx,dword ptr [esi+60h]
03ca0138 call    dword ptr ds:[3C20010h]
03ca013e mov     ecx,dword ptr [esi+58h]

0:000> ub 03ca0127
03ca0114 push    esi
03ca0115 push    ebx
03ca0116 sub     esp,1Ch

03ca0119 xor     eax,eax
03ca011b mov     dword ptr [ebp-18h],eax
03ca011e mov     dword ptr [ebp-28h],ecx
03ca0121 mov     dword ptr [ebp-24h],edx
03ca0124 mov     esi,dword ptr [ebp-28h]

0:000> ub 03ca0114
03ca0102 retf
03ca0103 add     eax,dword ptr [eax+36h]
03ca0106 retf
03ca0107 add     ebx,dword ptr [esi+esi-35h]
03ca010b add     esi,esp
03ca010d cmp     eax,8B550360h
03ca0112 in      al,dx
03ca0113 push    edi

From stack trace I suspected this code as JIT-compiled .NET code of the the main assemebly method. And indeed I found the similar call signatures like mine

03ca0141 cmp     dword ptr [ecx],ecx
03ca0143 call    dword ptr ds:[36067C0h]

in the following MSDN article:

Drill Into .NET Framework Internals to See How the CLR Creates Runtime Objects

Hence the name of this pattern: JIT Code.

- Dmitry Vostokov @ DumpAnalysis.org -

Sentinel Pointers

Wednesday, May 13th, 2009

Consider this crash point:

0:000> r
eax=02d0f15c ebx=02a62918 ecx=77e41c30 edx=00000000 esi=ffffffff edi=02a8ed28
eip=76154193 esp=02d0f124 ebp=02d0f130 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
Application!GetData+0xb:
76154193 8b9eac000000    mov     ebx,dword ptr [esi+0ACh] ds:0023:000000ab=????????

Seeing 000000ab address we can think that ESI was 0 but it was 0xFFFFFFFF. Adding 0xAC to it produced an effective NULL data pointer 0xAB through integer addition overflow if we consider addition as unsigned. It is easy to see the result if we consider 0xFFFFFFFF as signed -1. Looking at stack trace and function disassembly we see that 0xFFFFFFFF was passed as a parameter:

0:000> kv
ChildEBP RetAddr  Args to Child             
02d0f130 7616328d ffffffff 02d0f15c 02d0f150 Application!GetData+0xb
[…]
02d0ffec 00000000 740420d8 02a74070 00000000 kernel32!BaseThreadStart+0×34

0:000> u Application!GetData
Application!GetData:
76154188 mov     edi,edi
7615418a push    ebp
7615418b mov     ebp,esp
7615418d push    ecx
7615418e push    ebx
7615418f push    esi
76154190 mov     esi,dword ptr [ebp+8]
76154193 mov     ebx,dword ptr [esi+0ACh]

This is an example of a sentinel pointer marking the end of a linked list, for example, although NULL pointers having 0 value are usually used. Also -1 value can be used to assign an invalid pointer value. 

- Dmitry Vostokov @ DumpAnalysis.org -

Programming Language Pragmatics (3rd Edition)

Friday, May 8th, 2009

As soon as I wrote my review of the 2nd edition I found out that the 3rd edition was recently published and immediately bought it. I intend to read it from cover to cover again and publish my notes and comments in my reading notebook on Software Generalist blog. The new edition is also bundled with a companion CD.

Programming Language Pragmatics, Third Edition

Buy from Amazon

Hope in one of subsequent editions the author includes my Riemann Programming Language :-)

- Dmitry Vostokov @ DumpAnalysis.org -

NULL Data Pointer Pattern: case study

Wednesday, April 15th, 2009

Here is the promised case study for the previous post about data NULL pointers. The complete dump has this bugcheck:

0: kd> !analyze -v

[...]

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
This is a very common bugcheck.  Usually the exception address pinpoints the driver/function that caused the problem.  Always note this address as well as the link date of the driver/image that contains this address. Some common problems are exception code 0x80000003.  This means a hard coded breakpoint or assertion was hit, but this system was booted /NODEBUG.  This is not supposed to happen as developers should never have hardcoded breakpoints in retail code, but ... If this happens, make sure a debugger gets connected, and the system is booted /DEBUG.  This will let us see why this breakpoint is happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 8081c7c4, The address that the exception occurred at
Arg3: f1b5d730, Trap Frame
Arg4: 00000000

[...]

FAULTING_IP:
nt!IoIsOperationSynchronous+e
8081c7c4 f6412c02        test    byte ptr [ecx+2Ch],2

TRAP_FRAME:  f1b5d730 -- (.trap 0xfffffffff1b5d730)
[...]

0: kd> .trap 0xfffffffff1b5d730
ErrCode = 00000000
eax=8923b008 ebx=00000000 ecx=00000000 edx=8923b008 esi=891312d0 edi=89f0b300
eip=8081c7c4 esp=f1b5d7a4 ebp=f1b5d7a4 iopl=0 nv up ei ng nz ac pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010296
nt!IoIsOperationSynchronous+0xe:
8081c7c4 f6412c02   test    byte ptr [ecx+2Ch],2  ds:0023:0000002c=??

0: kd> kv 100
ChildEBP RetAddr  Args to Child             
f1b5d7a4 f42cdea9 8923b008 89f0b300 8923b008 nt!IoIsOperationSynchronous+0xe
f1b5d7bc 8081df85 89f0b300 8923b008 00000200 driveB!FsdDeviceIoControlFile+0×19
f1b5d7d0 808ed7a9 00000000 f1b5da84 f1b5db6c nt!IofCallDriver+0×45
f1b5da20 f3c3a521 89f0b300 f1b5da84 f1b5da84 nt!IoVolumeDeviceToDosName+0×89
WARNING: Stack unwind information not available. Following frames may be wrong.
f1b5da3c f3c3b58e 00000618 e4e00420 f1b5dad4 driverA+0×18531
[…]
f1b5dc3c 8081df85 89f48b48 87fa3008 89140d30 driverA+0×1df4

f1b5dc50 808f5437 87fa3078 89140d30 87fa3008 nt!IofCallDriver+0×45
f1b5dc64 808f61bf 89f48b48 87fa3008 89140d30 nt!IopSynchronousServiceTail+0×10b
f1b5dd00 808eed08 000000f0 00000000 00000000 nt!IopXxxControlFile+0×5e5
f1b5dd34 808897bc 000000f0 00000000 00000000 nt!NtDeviceIoControlFile+0×2a
f1b5dd34 7c8285ec 000000f0 00000000 00000000 nt!KiFastCallEntry+0xfc (TrapFrame @ f1b5dd64)
0856e154 7c826fcb 77e416f5 000000f0 00000000 ntdll!KiFastSystemCallRet
0856e158 77e416f5 000000f0 00000000 00000000 ntdll!NtDeviceIoControlFile+0xc
0856e1bc 6f050c6c 000000f0 5665824c 0856e234 kernel32!DeviceIoControl+0×137
[…]

From WDK help we know that the first parameter to IoIsOperationSynchronous is a pointer to an IRP structure:

0: kd> !irp 8923b008
Irp is active with 3 stacks 3 is current (= 0x8923b0c0)
 No Mdl: System buffer=878b7288: Thread 8758a020:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   
                     Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   
                     Args: 00000000 00000000 00000000 00000000
>[  e, 0]   0  0 89f0b300 00000000 00000000-00000000   
              \FileSystem\DriverB
                     Args: 00000200 00000000 004d0008 00000000

Disassembling the function shows some pointer dereferencing and we can reconstruct it starting from EBP+8, a pointer to an IRP. 

0: kd> .asm no_code_bytes
Assembly options: no_code_bytes

0: kd> u nt!IoIsOperationSynchronous nt!IoIsOperationSynchronous+0xe
nt!IoIsOperationSynchronous:
8081c7b6 mov     edi,edi
8081c7b8 push    ebp
8081c7b9 mov     ebp,esp
8081c7bb mov     eax,dword ptr [ebp+8]
8081c7be mov     ecx,dword ptr [eax+60h]
8081c7c1 mov     ecx,dword ptr [ecx+18h]

EAX+60 seems to be a current stack location member of IRP and it is a pointer itself to _IO_STACK_LOCATION structure:

0: kd> dt -r _IRP 8923b008
ntdll!_IRP
   +0x000 Type             : 6
   +0x002 Size             : 0x268
   +0x004 MdlAddress       : (null)
   +0x008 Flags            : 0x70
[...]
   +0x038 CancelRoutine    : (null)
   +0x03c UserBuffer       : 0xf1b5d814
   +0×040 Tail             : __unnamed
      +0×000 Overlay          : __unnamed
         +0×000 DeviceQueueEntry : _KDEVICE_QUEUE_ENTRY
         +0×000 DriverContext    : [4] (null)
         +0×010 Thread           : 0×8758a020 _ETHREAD
         +0×014 AuxiliaryBuffer  : (null)
         +0×018 ListEntry        : _LIST_ENTRY [ 0×0 - 0×0 ]
         +0×020 CurrentStackLocation : 0×8923b0c0 _IO_STACK_LOCATION
[…]

ECX+18 is a pointer to a file object in _IO_STACK_LOCATION structure:

0: kd> dt _IO_STACK_LOCATION 8923b008+60
ntdll!_IO_STACK_LOCATION
   +0x000 MajorFunction    : 0xc0 ''
   +0x001 MinorFunction    : 0xb0 ''
   +0x002 Flags            : 0x23 '#'
   +0x003 Control          : 0x89 ''
   +0x004 Parameters       : __unnamed
   +0x014 DeviceObject     : (null)
   +0×018 FileObject       : (null)
   +0×01c CompletionRoutine : (null)
   +0×020 Context          : (null)

2C offset at the crash point test byte ptr [ecx+2Ch],2 is _FILE_OBJECT Flags member:

0: kd> dt _FILE_OBJECT
ntdll!_FILE_OBJECT
   +0x000 Type             : Int2B
   +0x002 Size             : Int2B
   +0x004 DeviceObject     : Ptr32 _DEVICE_OBJECT
   +0x008 Vpb              : Ptr32 _VPB
   +0x00c FsContext        : Ptr32 Void
   +0x010 FsContext2       : Ptr32 Void
   +0x014 SectionObjectPointer : Ptr32 _SECTION_OBJECT_POINTERS
   +0x018 PrivateCacheMap  : Ptr32 Void
   +0x01c FinalStatus      : Int4B
   +0x020 RelatedFileObject : Ptr32 _FILE_OBJECT
   +0x024 LockOperation    : UChar
   +0x025 DeletePending    : UChar
   +0x026 ReadAccess       : UChar
   +0x027 WriteAccess      : UChar
   +0x028 DeleteAccess     : UChar
   +0x029 SharedRead       : UChar
   +0x02a SharedWrite      : UChar
   +0x02b SharedDelete     : UChar
   +0×02c Flags            : Uint4B
   +0×030 FileName         : _UNICODE_STRING
   +0×038 CurrentByteOffset : _LARGE_INTEGER
   +0×040 Waiters          : Uint4B
   +0×044 Busy             : Uint4B
   +0×048 LastLock         : Ptr32 Void
   +0×04c Lock             : _KEVENT
   +0×05c Event            : _KEVENT
   +0×06c CompletionContext : Ptr32 _IO_COMPLETION_CONTEXT

So it looks like driverA passed an IRP with NULL File object address to driverB and this is also shown in the output of !irp command above.

- Dmitry Vostokov @ DumpAnalysis.org -

Variable Kernel Stack in Vista and W2K8

Thursday, March 19th, 2009

Looking at one kernel memory dump from x64 Windows Server 2008 I noticed this API call (shown in blue):

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffffa60`138f4720 fffff800`01875f8a nt!KiSwapContext+0x7f
fffffa60`138f4860 fffff800`0187776a nt!KiSwapThread+0x2fa
fffffa60`138f48d0 fffff800`01ab16d6 nt!KeWaitForSingleObject+0x2da
fffffa60`138f4960 fffff800`01ab1667 nt!FsRtlCancellableWaitForMultipleObjects+0x62
fffffa60`138f49c0 fffffa60`06c515e0 nt!FsRtlCancellableWaitForSingleObject+0x27
fffffa60`138f4a00 fffffa60`06c611dc rdbss!RxWaitForStableCondition+0x11c
fffffa60`138f4a40 fffffa60`06c61c07 rdbss!RxFindOrCreateConnections+0x44c
fffffa60`138f4b20 fffffa60`06c56840 rdbss!RxConstructVirtualNetRoot+0xb7
fffffa60`138f4bc0 fffffa60`06c6381a rdbss!RxFindOrConstructVirtualNetRoot+0x594
fffffa60`138f4d30 fffffa60`06c54c42 rdbss!RxCreateTreeConnect+0x13e
fffffa60`138f4dc0 fffffa60`06c2fbf6 rdbss!RxCommonCreate+0x20a
fffffa60`138f4e80 fffffa60`06c5191a rdbss!RxFsdCommonDispatch+0x786
fffffa60`138f4f70 fffffa60`07e4f21f rdbss!RxFsdDispatch+0x21a
fffffa60`138f4fe0 fffffa60`011e05f5 mrxsmb!MRxSmbFsdDispatch+0xbf
fffffa60`138f5020 fffffa60`011e0130 mup!MupiCallUncProvider+0x159
fffffa60`138f5090 fffffa60`011e17af mup!MupStateMachine+0x120
fffffa60`138f50e0 fffffa60`00d200b4 mup!MupCreate+0x2c3
fffffa60`138f5160 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
[...]
3rd party filter drivers
[...]
fffffa60`138f55a0 fffff800`01aefa59 nt!IopParseDevice+0x5e3
fffffa60`138f5740 fffff800`01af3944 nt!ObpLookupObjectName+0x5eb
fffffa60`138f5850 fffff800`01affee0 nt!ObOpenObjectByName+0x2f4
fffffa60`138f5920 fffff800`01b00a0c nt!IopCreateFile+0x290
fffffa60`138f59c0 fffff800`0186fdf3 nt!NtCreateFile+0x78
fffffa60`138f5a50 fffff800`01870300 nt!KiSystemServiceCopyEnd+0x13
fffffa60`138f5c58 fffffa60`06c91a5e nt!KiServiceLinkage
fffffa60`138f5c60 fffff800`018913d1 dfsc!DfscConnOpenIpcConnectionCallout+0xbe
fffffa60`138f5d20 fffffa60`06c91d08 nt!KeExpandKernelStackAndCalloutEx+0×2e1
fffffa60`138f5db0 fffffa60`06c9bbcc dfsc!DfscGetIpcConnection+0×1f0
fffffa60`138f5e30 fffffa60`06c9bb21 dfsc!DfscRmGetReferral+0×78
fffffa60`138f5ea0 fffffa60`06c91470 dfsc!DfscGetDomainDCReferral+0×31
fffffa60`138f5ef0 fffffa60`06c917ec dfsc!DfscRmValidateDomainIterate+0×5c
fffffa60`138f5f40 fffffa60`06c915f5 dfsc!DfscValidateReferral+0xa0
fffffa60`138f5fb0 fffffa60`06c917ec dfsc!DfscRmValidateRootGetParent+0×75
fffffa60`138f5fe0 fffffa60`06c90825 dfsc!DfscValidateReferral+0xa0
fffffa60`138f6050 fffffa60`06c93905 dfsc!DfscCmValidateState+0×79
fffffa60`138f6090 fffffa60`06c9e759 dfsc!DfscSurrogateCreate+0×7d
fffffa60`138f6100 fffffa60`011e03ab dfsc!DfscSurrogatePreProcess+0xb9
fffffa60`138f6130 fffffa60`011e014f mup!MupCallSurrogatePrePost+0×10b
fffffa60`138f6190 fffffa60`011e17af mup!MupStateMachine+0×13f
fffffa60`138f61e0 fffffa60`00d200b4 mup!MupCreate+0×2c3
fffffa60`138f6260 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
[…]
3rd party filter drivers
[…]
fffffa60`138f6610 fffff800`01aefa59 nt!IopParseDevice+0×5e3
fffffa60`138f67b0 fffff800`01af3944 nt!ObpLookupObjectName+0×5eb
fffffa60`138f68c0 fffff800`01ac22f1 nt!ObOpenObjectByName+0×2f4
fffffa60`138f6990 fffff800`0186fdf3 nt!NtQueryAttributesFile+0×134
fffffa60`138f6c20 00000000`77285e4a nt!KiSystemServiceCopyEnd+0×13

This API is mentioned in the following presentation and document and can also be found in WDK:

PPT: Windows Memory Management Advances

DOC: Advances in Memory Management 

KeExpandKernelStackAndCallout

Its 3rd parameter is the stack size and we can see it used in disassembly where r8d register is used for 3rd parameter according to x64 calling convention and rcx is used for the first parameter, a function procedure to be executed with a guaranteed kernel stack size:

0: kd> kv 100
Child-SP          RetAddr           : Args to Child                                                           : Call Site
[...]
fffffa60`138f5c60 fffff800`018913d1 : 00000000`00000000 fffff880`10d6d3f8 00000000`00000000 00000000`00000000 : dfsc!DfscConnOpenIpcConnectionCallout+0xbe
fffffa60`138f5d20 fffffa60`06c91d08 : fffffa60`06c919a0 fffffa60`138f5df0 fffff880`102128d0 fffffa60`138f5f10 : nt!KeExpandKernelStackAndCalloutEx+0×2e1
fffffa60`138f5db0 fffffa60`06c9bbcc : 00000000`00000000 fffff880`10d6d3f8 00000000`00000000 fffff880`10d6d460 : dfsc!DfscGetIpcConnection+0×1f0
[…]

0: kd> ub fffffa60`06c91d08
dfsc!DfscGetIpcConnection+0×1c6:
fffffa60`06c91cde xor     r9d,r9d
fffffa60`06c91ce1 mov     qword ptr [rsp+50h],rax
fffffa60`06c91ce6 mov     rax,qword ptr [dfsc!DfscGlobalData+0×138 (fffffa60`06c8d758)]
fffffa60`06c91ced mov     r8d,6000h
fffffa60`06c91cf3 mov     qword ptr [rsp+40h],rdi
fffffa60`06c91cf8 mov     byte ptr [rsp+58h],r11b
fffffa60`06c91cfd mov     qword ptr [rsp+20h],rax
fffffa60`06c91d02 call    qword ptr [dfsc!_imp_KeExpandKernelStackAndCalloutEx (fffffa60`06c8b0d0)]

0: kd> ub fffffa60`06c91cde
dfsc!DfscGetIpcConnection+0x199:
fffffa60`06c91cb1 488b88b8000000  mov     rcx,qword ptr [rax+0B8h]
fffffa60`06c91cb8 0fba61100a      bt      dword ptr [rcx+10h],0Ah
fffffa60`06c91cbd 450f42df        cmovb   r11d,r15d
fffffa60`06c91cc1 488b4338        mov     rax,qword ptr [rbx+38h]
fffffa60`06c91cc5 488d542440      lea     rdx,[rsp+40h]
fffffa60`06c91cca 488d0dcffcffff  lea     rcx,[dfsc!DfscConnOpenIpcConnectionCallout (fffffa60`06c919a0)]
fffffa60`06c91cd1 4889442448      mov     qword ptr [rsp+48h],rax
fffffa60`06c91cd6 488d842490000000 lea     rax,[rsp+90h]

It is good sign to see it used in file system stacks because in the past the fixed kernel stacks resulted in stack overflows and double faults:

Stack Overflow Pattern (kernel mode)

- Dmitry Vostokov @ DumpAnalysis.org -

Review of Programming Language Pragmatics

Friday, March 6th, 2009

Every debugging engineer needs to know how the code is interpreted or compiled. Debugging complex problems or doing memory analysis on general-purpose operating systems often requires understanding the syntax and semantics of several programming languages and their run-time support. The knowledge of optimization techniques is also important for low-level debugging when the source code is not available. The following book provides an overview of all important concepts and discusses almost 50 languages. I read the first edition 6 years ago and I liked it so much that I’m now reading the second edition.

Programming Language Pragmatics, Second Edition

Buy from Amazon

- Dmitry Vostokov @ DumpAnalysis.org -

WDPF Book is #1 Assembly Language Bestseller

Monday, February 23rd, 2009

Looked this evening at Amazon and found that the book achieved #1 status (although it might not be the case at the time when you are reading this post):

#1 in  Books > Computers & Internet > Programming > Languages & Tools > Assembly Language Programming

- Dmitry Vostokov @ DumpAnalysis.org -

Windows Debugging book has been published!

Monday, February 2nd, 2009

I very proud to announce that after 3 weeks of final work the book has been released in both paperback and PDF format. In a week or so it should also appear on Amazon and other booksellers around the world. The book information and how to buy it can be found on the portal:

Windows Debugging: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org -

TOC for WDPF Book

Thursday, January 29th, 2009

Draft Table of Contents is available for the forthcoming Windows Debugging: Practical Foundations book to be released next week:

Draft Table of Contents

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg In Use: Debugging Exercises

Wednesday, December 24th, 2008

The analogy between learning a complex tool with its own language and a foreign natural language has been developed further after the release of WinDbg Learning Cards and finally culminated in “WinDbg In Use” book series with the first book to be published during the 1st quarter of 2009:

  • Title: WinDbg In Use: Debugging Exercises (Elementary and Intermediate Level)
  • Author: Dmitry Vostokov
  • Publisher: Opentask (15 March 2009)
  • Language: English
  • Product Dimensions: 23.5 x 19.1
  • ISBN-13: 978-1-906717-50-6
  • Paperback: 200 pages
  • Book Annotation: Includes 60 programmed exercises from real life debugging and crash dump analysis scenarios and multiple-choice questions with full answers, comments and suggestions for further reading.

Some example exercises will be published on this blog from time to time. I also plan a corresponding column in the forthcoming Debugged! magazine. 

- Dmitry Vostokov @ DumpAnalysis.org -

Debugged! Magazine

Tuesday, November 25th, 2008

As one of the new initiatives for the Year of Debugging  DumpAnalysis Portal will publish bimonthly full color 16 page publication called:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The only serial publication dedicated entirely to Windows® debugging

The first issue is planned for March, 2009 and will have ISBN-13: 978-1-906717-38-4. If it goes well I’m planning to have ISSN number assigned to it too. More details will be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org

MDAA Volume 2 is available on Amazon and B&N

Saturday, October 18th, 2008

Paperback edition of Memory Dump Analysis Anthology, Volume 2 is finally available on Amazon and Barnes & Noble. Search Inside is also available on Amazon. In addition, I updated the list of recommended books:

Listmania! Crash Dump Analysis and Debugging

Hardcover edition will be available on Amazon and B&N in 2-3 weeks.

- Dmitry Vostokov @ DumpAnalysis.org -

I’m Windows Internals certified!

Saturday, October 11th, 2008

Seems railroad to it was a success: just got this message in my e-mail:

Congratulations on passing your recent Microsoft Certification exam, inspiring confidence for your employer, your peers, and yourself with a widely-recognized validation of your skills on Microsoft technology.

Because I haven’t done any exam since Windows Internals beta I assumed that I passed it and I was right! After registering at Microsoft certification site as MCP I was able to build my logo:

Here is the link to Exam 70-660 information and required skills:

http://www.microsoft.com/learning/en/us/Exams/70-660.aspx

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 76)

Monday, October 6th, 2008

Most of the time Data Alignment manifests itself on Intel platforms from performance perspective and GP faults for some instructions that require natural boundary for their qword operands. There are no exceptions generally if we move a dword value from or to an odd memory location address when the whole operand fits into one page. However we need to take the possibility of page boundary spans into account when checking memory addresses for their validity. Consider this exception:

0: kd> .trap 0xffffffffa38df520
ErrCode = 00000002
eax=b6d9220f ebx=b6ab4ffb ecx=00000304 edx=eaf2fdea esi=b6d9214c edi=b6ab8189
eip=bfa10e6e esp=a38df594 ebp=a38df5ac iopl=0 nv up ei ng nz ac po cy
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000  efl=00010293
driver+0x2ae6e:
bfa10e6e 895304  mov    dword ptr [ebx+4],edx ds:0023:b6ab4fff=????????

The address seems to be valid:

0: kd> !pte b6ab4fff
               VA b6ab4fff
PDE at   C0300B68        PTE at C02DAAD0
contains 7F0DD863      contains 426B0863
pfn 7f0dd —DA–KWEV    pfn 426b0 —DA–KWEV

But careful examination of the instruction reveals that it writes 32 bit value so we need to inspect the next byte too because it is on another page:

0: kd> !pte b6ab4fff+1
               VA b6ab5000
PDE at   C0300B68        PTE at C02DAAD4
contains 7F0DD863      contains 00000080
pfn 7f0dd —DA–KWEV                           not valid
                       DemandZero
                       Protect: 4 - ReadWrite

Although the page is demand zero and this should have been satisfied by creating a new page filled with zeroes, my point here that the page could have been completely invalid or paged out in the case of IRQL >= 2. 

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump Analysis Anthology, Volume 2

Friday, October 3rd, 2008

“Everything is memory dump.”

I’m very excited to announce that Volume 2 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 2

In one or two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available by the end of October.

I’m often asked when Volume 3 is available and I currently plan to release it in October - November, 2009. In the mean time I’m planning to concentrate on other publishing projects. 

- Dmitry Vostokov @ DumpAnalysis.org -