Crash Dump Analysis Patterns (Part 9b)
This is a follow up to the previous Deadlock pattern post. In this part I’m going to show an example of ERESOURCE deadlock in the Windows kernel.
ERESOURCE (executive resource) is a Windows synchronization object that has ownership semantics.
An executive resource can be owned exclusively or can have a shared ownership. This is similar to the following file sharing analogy: when a file is opened for writing others can’t write or read it; if you have that file opened for reading others can read from it but can’t write to it.
ERESOURCE structure is linked into a list and have threads as owners which allows us to quickly find deadlocks using !locks command in kernel and complete memory dumps. Here is the definition of _ERESOURCE from x86 and x64 Windows:
0: kd> dt -r1 _ERESOURCE
+0x000 SystemResourcesList : _LIST_ENTRY
+0x000 Flink : Ptr32 _LIST_ENTRY
+0x004 Blink : Ptr32 _LIST_ENTRY
+0x008 OwnerTable : Ptr32 _OWNER_ENTRY
+0x000 OwnerThread : Uint4B
+0x004 OwnerCount : Int4B
+0x004 TableSize : Uint4B
+0x00c ActiveCount : Int2B
+0x00e Flag : Uint2B
+0x010 SharedWaiters : Ptr32 _KSEMAPHORE
+0x000 Header : _DISPATCHER_HEADER
+0x010 Limit : Int4B
+0x014 ExclusiveWaiters : Ptr32 _KEVENT
+0x000 Header : _DISPATCHER_HEADER
+0x018 OwnerThreads : [2] _OWNER_ENTRY
+0x000 OwnerThread : Uint4B
+0x004 OwnerCount : Int4B
+0x004 TableSize : Uint4B
+0x028 ContentionCount : Uint4B
+0x02c NumberOfSharedWaiters : Uint2B
+0x02e NumberOfExclusiveWaiters : Uint2B
+0x030 Address : Ptr32 Void
+0x030 CreatorBackTraceIndex : Uint4B
+0x034 SpinLock : Uint4B
0: kd> dt -r1 _ERESOURCE
nt!_ERESOURCE
+0x000 SystemResourcesList : _LIST_ENTRY
+0x000 Flink : Ptr64 _LIST_ENTRY
+0x008 Blink : Ptr64 _LIST_ENTRY
+0x010 OwnerTable : Ptr64 _OWNER_ENTRY
+0x000 OwnerThread : Uint8B
+0x008 OwnerCount : Int4B
+0x008 TableSize : Uint4B
+0x018 ActiveCount : Int2B
+0x01a Flag : Uint2B
+0x020 SharedWaiters : Ptr64 _KSEMAPHORE
+0x000 Header : _DISPATCHER_HEADER
+0x018 Limit : Int4B
+0x028 ExclusiveWaiters : Ptr64 _KEVENT
+0x000 Header : _DISPATCHER_HEADER
+0x030 OwnerThreads : [2] _OWNER_ENTRY
+0x000 OwnerThread : Uint8B
+0x008 OwnerCount : Int4B
+0x008 TableSize : Uint4B
+0x050 ContentionCount : Uint4B
+0x054 NumberOfSharedWaiters : Uint2B
+0x056 NumberOfExclusiveWaiters : Uint2B
+0x058 Address : Ptr64 Void
+0x058 CreatorBackTraceIndex : Uint8B
+0x060 SpinLock : Uint8B
If we have a list of resources from !locks output we can start following threads that own these resources. Owner threads are marked with a star character (*):
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks......
Resource @ 0x8815b928 Exclusively owned
Contention Count = 6234751
NumberOfExclusiveWaiters = 53
Threads: 89ab8db0-01<*>
Threads Waiting On Exclusive Access:
8810fa08 880f5b40 88831020 87e33020
880353f0 88115020 88131678 880f5db0
89295420 88255378 880f8b40 8940d020
880f58d0 893ee500 880edac8 880f8db0
89172938 879b3020 88091510 88038020
880407b8 88051020 89511db0 8921f020
880e9db0 87c33020 88064cc0 88044730
8803f020 87a2a020 89529380 8802d330
89a53020 89231b28 880285b8 88106b90
8803cbc8 88aa3020 88093400 8809aab0
880ea540 87d46948 88036020 8806e198
8802d020 88038b40 8826b020 88231020
890a2020 8807f5d0
We see that 53 threads are waiting for _KTHREAD 89ab8db0 to release _ERESOURCE 8815b928. Searching for this thread address reveals the following:
Resource @ 0x88159560 Exclusively owned
Contention Count = 166896
NumberOfExclusiveWaiters = 1
Threads: 8802a790-01<*>
Threads Waiting On Exclusive Access:
89ab8db0
We see that thread 89ab8db0 is waiting for 8802a790 to release resource 88159560. We continue searching for thread 8802a790 waiting for another thread but we skip occurences when this thread is not waiting:
Resource @ 0x881f7b60 Exclusively owned
Threads: 8802a790-01<*>
Resource @ 0x8824b418 Exclusively owned
Contention Count = 34
Threads: 8802a790-01<*>
Resource @ 0x8825e5a0 Exclusively owned
Threads: 8802a790-01<*>
Resource @ 0x88172428 Exclusively owned
Contention Count = 5
NumberOfExclusiveWaiters = 1
Threads: 8802a790-01<*>
Threads Waiting On Exclusive Access:
880f5020
Searching further we see that thread 8802a790 is waiting for thread 880f5020 to release resource 89bd7bf0:
Resource @ 0x89bd7bf0 Exclusively owned
Contention Count = 1
NumberOfExclusiveWaiters = 1
Threads: 880f5020-01<*>
Threads Waiting On Exclusive Access:
8802a790
If we look carefully we would see that we have already seen thread 880f5020 above and I repeat the fragment (with appropriate colors now):
Resource @ 0x88172428 Exclusively owned
Contention Count = 5
NumberOfExclusiveWaiters = 1
Threads: 8802a790-01<*>
Threads Waiting On Exclusive Access:
880f5020
We see that thread 880f5020 is waiting for thread 8802a790 and thread 8802a790 is waiting for thread 880f5020.
Therefore we have identified the classical deadlock. What we have to do now is to look at stack traces of these threads to see involved components.
- Dmitry Vostokov @ DumpAnalysis.org -
September 18th, 2008 at 8:41 am
[…] 원문링크: http://www.dumpanalysis.org/blog/index.php/2007/07/03/crash-dump-analysis-patterns-part-9b/ […]
October 25th, 2008 at 7:30 am
[…] space. Previously we discussed deadlock patterns involving critical sections in user space, executive resources in kernel space, mixed objects in user space and […]
February 17th, 2009 at 9:29 am
[…] Deadlock (executive resources) […]
September 9th, 2009 at 4:18 pm
[…] are some crash dump analysis pattern examples that involve executive resources: Deadlock (executive resources), High Contention (executive resources) and Wait Chain (executive […]
March 29th, 2010 at 11:02 am
[…] Today we introduce an icon for Deadlock (executive resources) pattern: […]