Crash Dump Analysis Patterns (Part 91)
Forthcoming Webinar on Pattern-Driven Software Diagnostics
2012 - The Year of Software Trace Analysis
Accelerated Memory Dump Analysis Training
Sponsored link: Memory Dump Analysis Services
Debugging Experts Magazine Online
Debugging Today Daily Newspaper
Sometimes we can observe rare events when abnormal conditions that usually result in a system crash result in a milder problem, for example, a service is unavailable and not affecting other services and users. It was reported that an application was freezing during user session logoff. A complete memory dump was saved at that time and its stack trace collection (!stacks command) shows the following suspicious thread in a user process (all other threads were normally waiting):
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[89cfa960 Application.exe]
ea0.001c4c 89a11db0 0499cd1 Blocked DriverA+0×69db
[…]
0: kd> !thread 89a11db0 16
THREAD 89a11db0 Cid 0ea0.1c4c Teb: 7ffdf000 Win32Thread: bc347a48 WAIT: (Unknown) KernelMode Non-Alertable
89b87770 Unknown
b97004ac NotificationEvent
IRP List:
899e2668: (0006,0244) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap daf62b28
Owning Process 89cfa960 Image: Application.exe
Attached Process N/A Image: N/A
Wait Start TickCount 909331 Ticks: 4824273 (0:20:56:19.265)
Context Switch Count 186 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.093
*** ERROR: Module load completed but symbols could not be loaded for Application.exe
Win32 Start Address Application (0×00406b2a)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init b60ceb30 Current b60cdf10 Base b60cf000 Limit b60cb000 Call b60ceb34
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr Args to Child
b60cdf28 80833485 89a11db0 00000002 00000000 nt!KiSwapContext+0×26
b60cdf54 808294b9 dc399008 89b87748 b60ce01c nt!KiSwapThread+0×2e5
b60cdf88 b96d69db 00000002 b60cdfbc 00000001 nt!KeWaitForMultipleObjects+0×3d7
WARNING: Stack unwind information not available. Following frames may be wrong.
b60cdfe8 b96d719e 89b87748 dc399008 b60ce01c DriverA+0×69db
[…]
We notice “89b87770 Unknown” and double check what object the thread is waiting for:
0: kd> dp b60cdfbc L00000002
b60cdfbc 89b87770 b97004ac
These are exactly the same objects that are listed in !thread command output. We see that the second one is normal and resides in a nonpaged area:
0: kd> dt _DISPATCHER_HEADER b97004ac
ntdll!_DISPATCHER_HEADER
+0x000 Type : 0 ''
+0x001 Absolute : 0 ''
+0x001 NpxIrql : 0 ''
+0x002 Size : 0x4 ''
+0x002 Hand : 0x4 ''
+0x003 Inserted : 0 ''
+0x003 DebugActive : 0 ''
+0x000 Lock : 262144
+0x004 SignalState : 0
+0x008 WaitListHead : _LIST_ENTRY [ 0x89a11e70 - 0x89a11e70 ]
0: kd> !address b97004ac
a71e3000 - 13e1d000
Usage KernelSpaceUsageNonPagedSystem
The other looks like an invalid Random Object from the free nonpaged pool entry (it even says about itself that it is bad) that used to belong in the past to Configuration Manager:
0: kd> !pool 89b87770
Pool page 89b87770 region is Nonpaged pool
[...]
89b87540 size: 98 previous size: 40 (Allocated) File (Protected)
*89b875d8 size: 260 previous size: 98 (Free) *CMpa
Pooltag CMpa : registry post apcs, Binary : nt!cm
89b87838 size: 28 previous size: 260 (Allocated) FSfm
[…]
0: kd> dd 89b87770
89b87770 bad0b0b0 00000000 00000000 00000000
89b87780 8a04be01 00000000 89b87788 89b87788
89b87790 00150006 e56c6946 8993e208 89ab96b8
89b877a0 00000000 00000000 bad0b0b0 c0000800
89b877b0 02110004 63426343 88ebbf80 00001000
89b877c0 00199000 00000000 8993e238 88d0d248
89b877d0 0019a000 00000000 00000000 00000000
89b877e0 00000000 00000000 00000000 00000000
0: kd> dt _DISPATCHER_HEADER 89b87770
ntdll!_DISPATCHER_HEADER
+0×000 Type : 0xb0 ”
+0×001 Absolute : 0xb0 ”
+0×001 NpxIrql : 0xb0 ”
+0×002 Size : 0xd0 ”
+0×002 Hand : 0xd0 ”
+0×003 Inserted : 0xba ”
+0×003 DebugActive : 0xba ”
+0×000 Lock : -1160728400
+0×004 SignalState : 0
+0×008 WaitListHead : _LIST_ENTRY [ 0×0 - 0×0 ]
Now some counterfactual thinking. One possible scenario after KeWaitForMultipleObjects was called to wait for both objects to become signalled (3rd WAIT_TYPE parameter) the free pool slot was allocated or coalesced with SignalState becoming nonzero by coincidence and other members becoming random values and then the second normal object becomes signalled when another thread sets the notification event…
- Dmitry Vostokov @ DumpAnalysis.org -
Sponsored link: Professional Software Debugging Services
/* Malware and Software Defects -> Victimware.org */
Copyright © 2006 - 2012. This is a non-profit research and scientific project.
_1125.png)
Citrix and Microsoft Customer Forum
Museum of Debugging and Memory Dumps
7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story
Crash and Hang Analysis Audit Service
CARE: Crash Analysis Report Environment
Crash Dump and Software Trace Analysis Training and Seminars
Access OpenTask Titles on Safari Books Online
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
International Memory Analysts and Debuggers Day: 07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA
AnnouncementsComing Soon:
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Management Bits: An Anthology from Reductionist Manager
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Introduction to Pattern-Driven Software Problem Solving
Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5
Windows Debugging Notebook: Essential User Space WinDbg Commands
Memory Dump Analysis Anthology, Volume 5
Memory Dump Analysis Anthology, Volume 4
Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
