Crash Dump Analysis Patterns (Part 91)
Sometimes we can observe rare events when abnormal conditions that usually result in a system crash result in a milder problem, for example, a service is unavailable and not affecting other services and users. It was reported that an application was freezing during user session logoff. A complete memory dump was saved at that time and its stack trace collection (!stacks command) shows the following suspicious thread in a user process (all other threads were normally waiting):
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[89cfa960 Application.exe]
ea0.001c4c 89a11db0 0499cd1 Blocked DriverA+0×69db
[…]
0: kd> !thread 89a11db0 16
THREAD 89a11db0 Cid 0ea0.1c4c Teb: 7ffdf000 Win32Thread: bc347a48 WAIT: (Unknown) KernelMode Non-Alertable
89b87770 Unknown
b97004ac NotificationEvent
IRP List:
899e2668: (0006,0244) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap daf62b28
Owning Process 89cfa960 Image: Application.exe
Attached Process N/A Image: N/A
Wait Start TickCount 909331 Ticks: 4824273 (0:20:56:19.265)
Context Switch Count 186 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.093
*** ERROR: Module load completed but symbols could not be loaded for Application.exe
Win32 Start Address Application (0×00406b2a)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init b60ceb30 Current b60cdf10 Base b60cf000 Limit b60cb000 Call b60ceb34
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr Args to Child
b60cdf28 80833485 89a11db0 00000002 00000000 nt!KiSwapContext+0×26
b60cdf54 808294b9 dc399008 89b87748 b60ce01c nt!KiSwapThread+0×2e5
b60cdf88 b96d69db 00000002 b60cdfbc 00000001 nt!KeWaitForMultipleObjects+0×3d7
WARNING: Stack unwind information not available. Following frames may be wrong.
b60cdfe8 b96d719e 89b87748 dc399008 b60ce01c DriverA+0×69db
[…]
We notice “89b87770 Unknown” and double check what object the thread is waiting for:
0: kd> dp b60cdfbc L00000002
b60cdfbc 89b87770 b97004ac
These are exactly the same objects that are listed in !thread command output. We see that the second one is normal and resides in a nonpaged area:
0: kd> dt _DISPATCHER_HEADER b97004ac
ntdll!_DISPATCHER_HEADER
+0x000 Type : 0 ''
+0x001 Absolute : 0 ''
+0x001 NpxIrql : 0 ''
+0x002 Size : 0x4 ''
+0x002 Hand : 0x4 ''
+0x003 Inserted : 0 ''
+0x003 DebugActive : 0 ''
+0x000 Lock : 262144
+0x004 SignalState : 0
+0x008 WaitListHead : _LIST_ENTRY [ 0x89a11e70 - 0x89a11e70 ]
0: kd> !address b97004ac
a71e3000 - 13e1d000
Usage KernelSpaceUsageNonPagedSystem
The other looks like an invalid Random Object from the free nonpaged pool entry (it even says about itself that it is bad) that used to belong in the past to Configuration Manager:
0: kd> !pool 89b87770
Pool page 89b87770 region is Nonpaged pool
[...]
89b87540 size: 98 previous size: 40 (Allocated) File (Protected)
*89b875d8 size: 260 previous size: 98 (Free) *CMpa
Pooltag CMpa : registry post apcs, Binary : nt!cm
89b87838 size: 28 previous size: 260 (Allocated) FSfm
[…]
0: kd> dd 89b87770
89b87770 bad0b0b0 00000000 00000000 00000000
89b87780 8a04be01 00000000 89b87788 89b87788
89b87790 00150006 e56c6946 8993e208 89ab96b8
89b877a0 00000000 00000000 bad0b0b0 c0000800
89b877b0 02110004 63426343 88ebbf80 00001000
89b877c0 00199000 00000000 8993e238 88d0d248
89b877d0 0019a000 00000000 00000000 00000000
89b877e0 00000000 00000000 00000000 00000000
0: kd> dt _DISPATCHER_HEADER 89b87770
ntdll!_DISPATCHER_HEADER
+0×000 Type : 0xb0 ”
+0×001 Absolute : 0xb0 ”
+0×001 NpxIrql : 0xb0 ”
+0×002 Size : 0xd0 ”
+0×002 Hand : 0xd0 ”
+0×003 Inserted : 0xba ”
+0×003 DebugActive : 0xba ”
+0×000 Lock : -1160728400
+0×004 SignalState : 0
+0×008 WaitListHead : _LIST_ENTRY [ 0×0 - 0×0 ]
Now some counterfactual thinking. One possible scenario after KeWaitForMultipleObjects was called to wait for both objects to become signalled (3rd WAIT_TYPE parameter) the free pool slot was allocated or coalesced with SignalState becoming nonzero by coincidence and other members becoming random values and then the second normal object becomes signalled when another thread sets the notification event…
- Dmitry Vostokov @ DumpAnalysis.org -