Swarm of shared locks, blocked threads and waiting time: pattern cooperation
CARE: Crash Analysis Report Environment
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
Here is further analysis of a memory dump used to illustrate Swarm of Shared Locks pattern. In that dump there were also exclusively held locks with many blocked threads:
Resource @ 0x8a04c408 Exclusively owned
Contention Count = 344875
NumberOfExclusiveWaiters = 6
Threads: 87eb3db0-01<*>
Threads Waiting On Exclusive Access:
88573db0 87b90378 86a49db0 891f4610
8662b020 87127db0
Resource @ 0x89678e80 Exclusively owned
Contention Count = 10261
NumberOfExclusiveWaiters = 2
Threads: 87eb3db0-01<*>
Threads Waiting On Exclusive Access:
89131bf0 88d12db0
Resource @ 0x88d099d8 Exclusively owned
Contention Count = 562811
NumberOfExclusiveWaiters = 4
Threads: 873611d8-01<*>
Threads Waiting On Exclusive Access:
88b8bb88 88a72c50 89359af0 88d865e8
Resource @ 0x86db9248 Exclusively owned
Contention Count = 1382269
NumberOfSharedWaiters = 2
NumberOfExclusiveWaiters = 11
Threads: 86ab2020-01<*> 8769cdb0-01 880c77b8-01
Threads Waiting On Exclusive Access:
87bf4020 890dc020 874c01c0 884ef020
86913af8 875bab10 88e8a0d8 8923cdb0
894eca18 86aa6830 86f293a8
Resource @ 0x873a88d0 Exclusively owned
Contention Count = 719758
NumberOfExclusiveWaiters = 8
Threads: 88d5f990-01<*>
Threads Waiting On Exclusive Access:
8759ea88 871b6db0 88117710 87cb4718
883eb638 87239020 881ad020 891b9188
Resource @ 0x88c379a0 Exclusively owned
Contention Count = 126686
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 8
Threads: 882b8020-01<*> 88951520-01
Threads Waiting On Exclusive Access:
877d34a8 8939fdb0 87fc5668 8851fdb0
86fad850 87f1f450 8a1749f0 876a78d0
Resource @ 0x88ca9250 Exclusively owned
Contention Count = 319721
NumberOfExclusiveWaiters = 4
Threads: 88607908-01<*>
Threads Waiting On Exclusive Access:
86829370 892ae8e8 87205208 87b6d7e0
Resource @ 0x86a90ef8 Exclusively owned
Contention Count = 852830
NumberOfExclusiveWaiters = 12
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
88a9c9b0 88a50db0 87117928 890e4c50
874ffb30 88b540f8 8705d020 8687edb0
87143188 8703e430 885b6aa0 8842bc50
Resource @ 0x88954538 Exclusively owned
Contention Count = 40708
NumberOfExclusiveWaiters = 1
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
878ee980
Resource @ 0x88617eb8 Exclusively owned
Contention Count = 43531
NumberOfExclusiveWaiters = 2
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
88851db0 87382c50
Resource @ 0x87288bc8 Exclusively owned
Contention Count = 644675
NumberOfExclusiveWaiters = 2
Threads: 874e4508-01<*>
Threads Waiting On Exclusive Access:
88863b08 89479650
Resource @ 0x87c3d8b0 Exclusively owned
Contention Count = 335064
NumberOfExclusiveWaiters = 8
Threads: 87f44520-01<*>
Threads Waiting On Exclusive Access:
88277190 88eceb48 87f0d308 8694d460
88461db0 876734a8 871721b0 88c2adb0
All threads owning various locks exclusively are stuck in processing page fault code, for example:
0: kd> !thread 87eb3db0 1f
THREAD 87eb3db0 Cid 47ac.57c8 Teb: 7ffd7000 Win32Thread: bc151230 WAIT: (Unknown) KernelMode Non-Alertable
8743e4e0 NotificationEvent
IRP List:
8660c900: (0006,0094) Flags: 00000900 Mdl: 00000000
Not impersonating
DeviceMap e1003890
Owning Process 88e49918 Image: csrss.exe
Wait Start TickCount 15420972 Ticks: 2527 (0:00:00:39.484)
Context Switch Count 1430991 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:02.734
Start Address 0×75a8e96c
Stack Init a3cf7000 Current a3cf6430 Base a3cf7000 Limit a3cf4000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr
a3cf6448 8083d5b1 nt!KiSwapContext+0×26
a3cf6474 8083df9e nt!KiSwapThread+0×2e5
a3cf64bc 8082629e nt!KeWaitForSingleObject+0×346
a3cf64e4 80826480 nt!MiWaitForInPageComplete+0×1f
a3cf656c 8084790e nt!MiDispatchFault+0xda3
a3cf65c8 80836c2a nt!MmAccessFault+0×64a
a3cf65c8 bfa38de0 nt!KiTrap0E+0xdc (TrapFrame @ a3cf65e0)
a3cf6a24 bf854a72 win32k!vSetPointer+0×36f
a3cf6a50 bf8b1b74 win32k!GreSetPointer+0×66
a3cf6a7c bf883183 win32k!zzzUpdateCursorImage+0×1cc
a3cf6a8c bf884b06 win32k!zzzSetFMouseMoved+0xd5
a3cf6ad4 bf81530a win32k!ProcessQueuedMouseEvents+0×1c4
a3cf6d30 bf86fd25 win32k!RawInputThread+0×5b4
a3cf6d40 bf898a52 win32k!xxxCreateSystemThreads+0×60
a3cf6d54 80833bef win32k!NtUserCallOneParam+0×23
a3cf6d54 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ a3cf6d64)
0: kd> !thread 87571640 1f
THREAD 87571640 Cid 49f4.65b4 Teb: 7ffdf000 Win32Thread: bc011680 WAIT: (Unknown) KernelMode Non-Alertable
8870db90 NotificationEvent
Not impersonating
DeviceMap e24f6570
Owning Process 87be4a00 Image: ApplicationC.EXE
Wait Start TickCount 15420974 Ticks: 2525 (0:00:00:39.453)
Context Switch Count 25640 LargeStack
UserTime 00:00:00.921
KernelTime 00:00:03.859
Win32 Start Address 0×30002658
Start Address 0×77e617f8
Stack Init 9a318600 Current 9a317b70 Base 9a319000 Limit 9a314000 Call 9a31860c
Priority 14 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
9a317b88 8083d5b1 nt!KiSwapContext+0×26
9a317bb4 8083df9e nt!KiSwapThread+0×2e5
9a317bfc 8082629e nt!KeWaitForSingleObject+0×346
9a317c24 80826480 nt!MiWaitForInPageComplete+0×1f
9a317cac 8084790e nt!MiDispatchFault+0xda3
9a317d08 80836c2a nt!MmAccessFault+0×64a
9a317d08 bf8b5485 nt!KiTrap0E+0xdc (TrapFrame @ 9a317d20)
9a317db4 bf8b526c win32k!vSolidFillRect1+0xb0
9a317f58 bf8ad7d2 win32k!vDIBSolidBlt+0×102
9a317fc4 bfa285d1 win32k!EngBitBlt+0xe1
9a3180c8 bf899b57 win32k!GrePatBltLockedDC+0×1ea
9a318160 bf8b32bb win32k!GrePolyPatBltInternal+0×17c
9a31819c bf8bd71c win32k!GrePolyPatBlt+0×45
9a31822c bf85e3d5 win32k!DrawEdge+0×23a
9a318274 bf8ae338 win32k!xxxDrawWindowFrame+0×170
9a3182d4 bf8847d1 win32k!xxxRealDefWindowProc+0×7a7
9a3182ec bf884801 win32k!xxxWrapRealDefWindowProc+0×16
9a318308 bf8c1769 win32k!NtUserfnDWORD+0×27
9a318340 80833bef win32k!NtUserMessageCall+0xc0
9a318340 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ 9a318364)
We also see that their waiting time is almost the same, 39 seconds. This means that the problem with paging probably started at that time before the crash dump was forced.
- Dmitry Vostokov @ DumpAnalysis.org -
_1125.png)
Coming Soon:
Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
