Inconsistent dump, stack trace collection, LPC, thread, process, executive resource wait chains, missing threads and waiting thread time: pattern cooperation
Forthcoming Webinar on Pattern-Driven Software Diagnostics
2012 - The Year of Software Trace Analysis
Accelerated Memory Dump Analysis Training
Sponsored link: Memory Dump Analysis Services
Debugging Experts Magazine Online
Debugging Today Daily Newspaper
Here is a synthetic case study to show various wait chain patterns. The complete memory dump from a frozen system is inconsistent, saved by LiveKd. Stack trace collection shows many threads waiting for LPC replies:
THREAD 87432520 Cid 0b10.0dd8 Teb: 7ff83000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
8743270c Semaphore Limit 0x1
Waiting for reply to LPC MessageId 007ade85:
Current LPC port d676e560
Not impersonating
DeviceMap d6f05be8
Owning Process 87581c78 Image: ProcessA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 4093415 Ticks: 659565 (0:02:51:45.703)
Context Switch Count 111255
UserTime 00:00:27.781
KernelTime 00:00:02.015
Win32 Start Address MSVCR71!_threadstartex (0×7c3494f6)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b6439000 Current b6438c08 Base b6439000 Limit b6436000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
b6438c10 00000000 0×0
Checking MessageId by using !lpc command gives us the following LPC server thread that is waiting for a mutant owned by thread 866d63e8 (this is equivalent that thread 85b209d0 is waiting for thread 866d63e8):
THREAD 85b209d0 Cid 1524.2770 Teb: 7ff78000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
85e9ba00 Mutant - owning thread 866d63e8
Not impersonating
DeviceMap d64008d8
Owning Process 87200880 Image: ProcessB.exe
Attached Process N/A Image: N/A
Wait Start TickCount 4093415 Ticks: 659565 (0:02:51:45.703)
Context Switch Count 28
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0×007ade85
LPC Server thread working on message Id 7ade85
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b42e5000 Current b42e4c60 Base b42e5000 Limit b42e2000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr
b42e4c68 00000000 0xa000a02
Thread 866d63e8 is waiting for a process 86b33b30:
THREAD 866d63e8 Cid 1524.3984 Teb: 7ff6e000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
86b33b30 ProcessObject
866d6460 NotificationTimer
Not impersonating
DeviceMap d64008d8
Owning Process 87200880 Image: ProcessB.exe
Attached Process N/A Image: N/A
Wait Start TickCount 4755080
Context Switch Count 4767
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0×007a5051
LPC Server thread working on message Id 7a5051
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init ab459000 Current ab458c60 Base ab459000 Limit ab456000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
ab458c78 8083d5b1 nt!KiSwapContext+0×26
ab458ca4 8083df9e nt!KiSwapThread+0×2e5
ab458cec 8092ae67 nt!KeWaitForSingleObject+0×346
ab458d50 80833bef nt!NtWaitForSingleObject+0×9a
ab458d50 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ ab458d64)
0499fbec 7c827d29 ntdll!KiFastSystemCallRet
0499fbf0 76548721 ntdll!ZwWaitForSingleObject+0xc
0499fc0c 7654aa54 ProcessB!WaitForProcess+0×4a
[…]
0499ffec 00000000 kernel32!BaseThreadStart+0×34
We find the following thread in the process 86b33b30 (there is only one thread left where we expect many of them in ProcessC):
THREAD 85a6fdb0 Cid 5c0c.26f4 Teb: 7ffdf000 Win32Thread: b9b778a0 WAIT: (Unknown) KernelMode Non-Alertable
86686030 SynchronizationEvent
85a6fe28 NotificationTimer
Not impersonating
DeviceMap d6767248
Owning Process 86b33b30 Image: ProcessC.EXE
Attached Process N/A Image: N/A
Wait Start TickCount 4755681
Context Switch Count 77668 LargeStack
UserTime 00:00:00.437
KernelTime 00:00:04.421
*** ERROR: Module load completed but symbols could not be loaded for ProcessC.EXE
Win32 Start Address 0×300010cc
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init 9ad92000 Current 9ad91a40 Base 9ad92000 Limit 9ad8d000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
9ad91a58 8083d5b1 nt!KiSwapContext+0×26
9ad91a84 8083df9e nt!KiSwapThread+0×2e5
9ad91acc 8081e05b nt!KeWaitForSingleObject+0×346
9ad91b08 8082e012 nt!ExpWaitForResource+0xd5
9ad91b28 b6a9c1ee nt!ExAcquireResourceExclusiveLite+0×8d
WARNING: Stack unwind information not available. Following frames may be wrong.
9ad91b38 b6ab09d3 DriverA+0×21ee
[…]
9ad91c3c 80840153 DriverA+0×1ed6
9ad91c50 8092b51f nt!IofCallDriver+0×45
9ad91c64 8092b454 nt!IopSynchronousServiceTail+0×10b
9ad91d00 8092b574 nt!IopXxxControlFile+0×60f
9ad91d34 80833bef nt!NtDeviceIoControlFile+0×2a
9ad91d34 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ 9ad91d64)
0012d4e0 00000000 ntdll!KiFastSystemCallRet
It is blocked by DriverA waiting for an executive resource. Fortunately !locks command works for this inconsistent dump and we finally reach thread 86ba5638:
0: kd> !locks
Resource @ 0x85610d30 Exclusively owned
Contention Count = 4
NumberOfExclusiveWaiters = 1
Threads: 86ba5638-01<*>
Threads Waiting On Exclusive Access:
85a6fdb0
This thread belongs to another instance of ProcessC.exe (different process 8690dd88):
0: kd> !thread 86ba5638 1f
THREAD 86ba5638 Cid 186c.1574 Teb: 7ffdf000 Win32Thread: b9a28a70 WAIT: (Unknown) KernelMode Non-Alertable
8944e3d4 SynchronizationEvent
Not impersonating
DeviceMap d6767248
Owning Process 8690dd88 Image: ProcessC.EXE
Attached Process N/A Image: N/A
Wait Start TickCount 4074148 Ticks: 678832 (0:02:56:46.750)
Context Switch Count 95896 LargeStack
UserTime 00:00:00.593
KernelTime 00:00:05.343
*** ERROR: Module load completed but symbols could not be loaded for ProcessC.EXE
Win32 Start Address 0×300010cc
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init 99ef2000 Current 99ef19c0 Base 99ef2000 Limit 99eec000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
99ef19d8 8083d5b1 nt!KiSwapContext+0×26
99ef1a04 8083df9e nt!KiSwapThread+0×2e5
99ef1a4c 80853f3f nt!KeWaitForSingleObject+0×346
99ef1a64 8081d45f nt!KiAcquireFastMutex+0×13
99ef1a70 b6a9c70d nt!ExAcquireFastMutex+0×20
WARNING: Stack unwind information not available. Following frames may be wrong.
99ef1a7c b6aaf22a DriverA+0×270d
[…]
99ef1c3c 80840153 DriverA+0×1ed6
99ef1c50 8092b51f nt!IofCallDriver+0×45
99ef1c64 8092b454 nt!IopSynchronousServiceTail+0×10b
99ef1d00 8092b574 nt!IopXxxControlFile+0×60f
99ef1d34 80833bef nt!NtDeviceIoControlFile+0×2a
99ef1d34 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ 99ef1d64)
0012db08 00000000 ntdll!KiFastSystemCallRet
We see this thread is also blocked by DriverA. We also check thread waiting times. All threads involved in wait chains have their Ticks value less than 86ba5638. This means that thread 86ba5638 was blocked earlier. We contact DriverA vendor for any possible updates.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sponsored link: Professional Software Debugging Services
/* Malware and Software Defects -> Victimware.org */
Copyright © 2006 - 2012. This is a non-profit research and scientific project.
_1125.png)
Citrix and Microsoft Customer Forum
Museum of Debugging and Memory Dumps
7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story
Crash and Hang Analysis Audit Service
CARE: Crash Analysis Report Environment
Crash Dump and Software Trace Analysis Training and Seminars
Access OpenTask Titles on Safari Books Online
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
International Memory Analysts and Debuggers Day: 07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA
AnnouncementsComing Soon:
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Management Bits: An Anthology from Reductionist Manager
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Introduction to Pattern-Driven Software Problem Solving
Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5
Windows Debugging Notebook: Essential User Space WinDbg Commands
Memory Dump Analysis Anthology, Volume 5
Memory Dump Analysis Anthology, Volume 4
Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
