Crash Dump Analysis Patterns (Part 96)
For certain stack traces we should always be aware of Coincidental Frames similar to Coincidental Symbolic Information pattern for raw stack data. Such frames can lead to a wrong analysis conclusion. Consider this stack trace fragment from a kernel memory dump:
0: kd> kL 100
ChildEBP RetAddr
9c5b6550 8082d9a4 nt!KeBugCheckEx+0×1b
9c5b6914 8088befa nt!KiDispatchException+0×3a2
9c5b697c 8088beae nt!CommonDispatchException+0×4a
9c5b699c 80a6056d nt!KiExceptionExit+0×186
9c5b69a0 80893ae2 hal!KeReleaseQueuedSpinLock+0×2d
9c5b6a08 b20c3de5 nt!MiFreePoolPages+0×7dc
WARNING: Stack unwind information not available. Following frames may be wrong.
9c5b6a48 b20c4107 DeriverA+0×17de5
[…]
The frame with MiFreePoolPages symbol might suggest some sort of a pool corruption. We can even double check return addresses and see the valid common sense assembly language code:
0: kd> ub 8088beae
nt!KiExceptionExit+0×167:
8088be8f 33c9 xor ecx,ecx
8088be91 e81a000000 call nt!CommonDispatchException (8088beb0)
8088be96 33d2 xor edx,edx
8088be98 b901000000 mov ecx,1
8088be9d e80e000000 call nt!CommonDispatchException (8088beb0)
8088bea2 33d2 xor edx,edx
8088bea4 b902000000 mov ecx,2
8088bea9 e802000000 call nt!CommonDispatchException (8088beb0)
0: kd> ub 80a6056d
hal!KeReleaseQueuedSpinLock+0×1b:
80a6055b 7511 jne hal!KeReleaseQueuedSpinLock+0×2e (80a6056e)
80a6055d 50 push eax
80a6055e f00fb119 lock cmpxchg dword ptr [ecx],ebx
80a60562 58 pop eax
80a60563 7512 jne hal!KeReleaseQueuedSpinLock+0×37 (80a60577)
80a60565 5b pop ebx
80a60566 8aca mov cl,dl
80a60568 e8871e0000 call hal!KfLowerIrql (80a623f4)
0: kd> ub 80893ae2
nt!MiFreePoolPages+0×7c3:
80893ac9 761c jbe nt!MiFreePoolPages+0×7e1 (80893ae7)
80893acb ff75f8 push dword ptr [ebp-8]
80893ace ff7508 push dword ptr [ebp+8]
80893ad1 e87ea1fcff call nt!MiFreeNonPagedPool (8085dc54)
80893ad6 8a55ff mov dl,byte ptr [ebp-1]
80893ad9 6a0f push 0Fh
80893adb 59 pop ecx
80893adc ff1524118080 call dword ptr [nt!_imp_KeReleaseQueuedSpinLock (80801124)]
0: kd> ub b20c3de5
DriverA+0×17dcf:
b20c3dcf 51 push ecx
b20c3dd0 ff5010 call dword ptr [eax+10h]
b20c3dd3 eb10 jmp DriverA+0×17de5 (b20c3de5)
b20c3dd5 8b5508 mov edx,dword ptr [ebp+8]
b20c3dd8 52 push edx
b20c3dd9 8d86a0000000 lea eax,[esi+0A0h]
b20c3ddf 50 push eax
b20c3de0 e8ebf1ffff call DriverA+0×16fd0 (b20c2fd0)
However, if we try to reconstruct the stack trace manually we would naturally skip these 3 frames (shown in magenta):
9c5b6550 8082d9a4 nt!KeBugCheckEx+0x1b
9c5b6914 8088befa nt!KiDispatchException+0x3a2
9c5b697c 8088beae nt!CommonDispatchException+0x4a
9c5b699c 80a6056d nt!KiExceptionExit+0×186
9c5b69a0 80893ae2 hal!KeReleaseQueuedSpinLock+0×2d
9c5b6a08 b20c3de5 nt!MiFreePoolPages+0×7dc
9c5b6a48 b20c4107 DeriverA+0×17de5
[…]
0: kd> !thread
THREAD 8f277020 Cid 081c.7298 Teb: 7ff11000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
8e234b60: (0006,0094) Flags: 00000000 Mdl: 00000000
Not impersonating
DeviceMap e1002880
Owning Process 8fc78b80 Image: ProcessA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 49046879 Ticks: 0
Context Switch Count 10
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address DllA!ThreadA (0x7654dc90)
Start Address kernel32!BaseThreadStartThunk (0x77e617dc)
Stack Init 9c5b7000 Current 9c5b6c50 Base 9c5b7000 Limit 9c5b4000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr Args to Child
[…]
0: kd> dds 9c5b4000 9c5b7000
9c5b4000 00000000
9c5b4004 00000000
9c5b4008 00000000
[...]
9c5b6290 ffdff13c
9c5b6294 9c5b6550
9c5b6298 80827e01 nt!KeBugCheckEx+0×1b
9c5b629c 00000008
9c5b62a0 00000286
[…]
9c5b654c 00000000
9c5b6550 9c5b6914
9c5b6554 8082d9a4 nt!KiDispatchException+0×3a2
9c5b6558 0000008e
9c5b655c c0000005
[…]
9c5b6910 ffffffff
9c5b6914 9c5b6984
9c5b6918 8088befa nt!CommonDispatchException+0×4a
9c5b691c 9c5b6930
9c5b6920 00000000
[…]
9c5b6980 8088beae nt!KiExceptionExit+0×186
9c5b6984 9c5b6a08
9c5b6988 b20c3032 DriverA+0×17032
9c5b698c badb0d00
9c5b6990 00000006
9c5b6994 8dc11cec
9c5b6998 808b6900 nt!KiTimerTableLock+0×3c0
9c5b699c 9c5b69d4
9c5b69a0 80a6056d hal!KeReleaseQueuedSpinLock+0×2d
9c5b69a4 80893ae2 nt!MiFreePoolPages+0×7dc
9c5b69a8 808b0b40 nt!NonPagedPoolDescriptor
9c5b69ac 03151fd0
9c5b69b0 00000000
9c5b69b4 00000000
[…]
9c5b6a04 8f47123b
9c5b6a08 9c5b6a48
9c5b6a0c b20c3de5 DriverA+0×17de5
9c5b6a10 8e3640a0
9c5b6a14 8f4710d0
[…]
9c5b6a44 00000000
9c5b6a48 9c5b6a80
9c5b6a4c b20c4107 DriverA+0×18107
9c5b6a50 8f4710d0
9c5b6a54 9c5b6a6c
[…]
If we try to find a pointer to the exception record we get this crash address:
0: kd> .exr 9c5b6930
ExceptionAddress: b20c3032 (DriverA+0×00017032)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000157
Attempt to read from address 00000157
If we disassemble it we see an inlined string or memory copy, perhaps wcscpy function:
0: kd> u b20c3032
DriverA+0×17032:
b20c3032 f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
b20c3034 8bcb mov ecx,ebx
b20c3036 83e103 and ecx,3
b20c3039 f3a4 rep movs byte ptr es:[edi],byte ptr [esi]
b20c303b 8b750c mov esi,dword ptr [ebp+0Ch]
b20c303e 0fb7ca movzx ecx,dx
b20c3041 894e14 mov dword ptr [esi+14h],ecx
b20c3044 8b700c mov esi,dword ptr [eax+0Ch]
So the problem happened in DriverA code, not in MiFreePoolPages or KeReleaseQueuedSpinLock.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
May 10th, 2015 at 11:46 am
This often happens when we have some “boundary” such as in Exception Stack Trace pattern.