Now I resume previously introduced memory analysis succession patterns post series. In this part I introduce a case study where Wait Chains (executive resources) and Swarm of Shared Locks were probably resulted from a Spiking Thread. We have these resource locks:
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
Resource @ 0x808a6860 Shared 9 owning threads
Threads: 8e739b40-01<*> 8e07db58-01<*> 8e455d18-01<*> 8df72958-01<*>
8e356620-01<*> 8d7e9700-01<*> 8e73a660-01<*> 8dbb0af0-01<*>
8e01c9b8-01<*>
Resource @ 0xf7b6d5b0 Shared 2 owning threads
Contention Count = 1
Threads: 8e741898-01<*> 8e73fb40-01<*>
Resource @ 0x8e5a1534 Shared 12 owning threads
Contention Count = 76
NumberOfSharedWaiters = 7
NumberOfExclusiveWaiters = 1
Threads: 8e73fb40-01 8d80fc70-01<*> 8dc226c0-01<*> 8dc84db0-01<*>
8e416458-01<*> 8dbf1630-01<*> 8e740db0-01<*> 8d7e9700-01<*>
8d818420-01<*> 8e7413b8-01<*> 8e739020-01 8d80ac70-01<*>
8e30ec88-01<*> 8e7408d0-01 8dd022a8-01 8e59a2f8-01
8dc4d300-01 8dc36278-01<*> 8e060078-01
Threads Waiting On Exclusive Access:
8d818870
Resource @ 0x8e5a1368 Shared 3 owning threads
Contention Count = 132
NumberOfExclusiveWaiters = 9
Threads: 8d7c5370-01<*> 8e416458-01<*> 8e72f480-01<*>
Threads Waiting On Exclusive Access:
8d7e9700 8d80ac70 8e741b08 8dc84db0
8df72958 8e73e8d0 8dbe0388 8e7413b8
8e741898
Resource @ 0x8e74a3b0 Exclusively owned
Contention Count = 11
NumberOfSharedWaiters = 2
NumberOfExclusiveWaiters = 1
Threads: 8dbe0388-01<*> 8e73e660-01 8e740020-01
Threads Waiting On Exclusive Access:
8d80fc70
Resource @ 0x8e54f810 Exclusively owned
Contention Count = 118
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 2
Threads: 8e72f480-01<*> 8e73f8d0-01
Threads Waiting On Exclusive Access:
8d7c5370 8e416458
Resource @ 0x8e6db008 Shared 1 owning threads
Threads: 8e73f8d0-01<*>
Resource @ 0x8e75e3c0 Exclusively owned
Threads: 8d7e9700-01<*>
Resource @ 0x8e6f14b0 Exclusively owned
Contention Count = 7
NumberOfSharedWaiters = 2
Threads: 8d80fc70-01<*> 8dbf1630-01 8e73f3f0-01
Resource @ 0x8e707618 Exclusively owned
Threads: 8dc84db0-01<*>
Resource @ 0x8e6c1780 Shared 1 owning threads
Contention Count = 3
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 1
Threads: 8e741b08-01<*> 8e73fdb0-01
Threads Waiting On Exclusive Access:
8dc36278
Resource @ 0x8e1fa370 Exclusively owned
Threads: 8df72958-01<*>
Resource @ 0x8e290b38 Exclusively owned
Threads: 8df72958-01<*>
Resource @ 0x8e692be0 Shared 1 owning threads
Contention Count = 12
Threads: 8df72958-01<*>
Resource @ 0x8e3a03e0 Exclusively owned
Contention Count = 4
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 1
Threads: 8dbe0388-01<*> 8e740660-01
Threads Waiting On Exclusive Access:
8e30ec88
Resource @ 0x8e33d6a8 Exclusively owned
Contention Count = 1
Threads: 8dc36278-01<*>
Resource @ 0x8e33d640 Exclusively owned
Contention Count = 1
Threads: 8dc36278-01<*>
Resource @ 0x8e17f890 Exclusively owned
Contention Count = 2
NumberOfExclusiveWaiters = 1
Threads: 8dc84db0-01<*>
Threads Waiting On Exclusive Access:
8e740db0
Resource @ 0x8e17f828 Exclusively owned
Threads: 8dc84db0-01<*>
Resource @ 0x8e09fc40 Exclusively owned
Contention Count = 3
NumberOfSharedWaiters = 2
Threads: 8e416458-01<*> 8e076770-01 8dbf5b70-01
Resource @ 0x8e09fbd8 Exclusively owned
Threads: 8e416458-01<*>
Resource @ 0x8df021a0 Exclusively owned
Threads: 8d7e9700-01<*>
Resource @ 0x8dffce50 Exclusively owned
Contention Count = 2
NumberOfExclusiveWaiters = 1
Threads: 8dbf1630-01<*>
Threads Waiting On Exclusive Access:
8dc226c0
Resource @ 0x8df147f8 Exclusively owned
Contention Count = 4
NumberOfSharedWaiters = 1
Threads: 8dbf1630-01<*> 8e7403f0-01
Resource @ 0x8e599de8 Exclusively owned
Threads: 8d7c5370-01<*>
Resource @ 0x8e324ee8 Shared 1 owning threads
Contention Count = 1
Threads: 8d7c5370-01<*>
Resource @ 0x8e21bc60 Exclusively owned
Threads: 8dbe0388-01<*>
Resource @ 0x8e446f18 Exclusively owned
Threads: 8d80fc70-01<*>
Resource @ 0x8df65570 Shared 1 owning threads
Contention Count = 3
Threads: 8dc5f698-01<*>
Resource @ 0x8e085100 Exclusively owned
Contention Count = 1
Threads: 8e73e8d0-01<*>
Resource @ 0x8dbe51b8 Exclusively owned
Threads: 8dbe0388-01<*>
Resource @ 0x8df458c8 Exclusively owned
Contention Count = 1
NumberOfExclusiveWaiters = 1
Threads: 8d80ac70-01<*>
Threads Waiting On Exclusive Access:
8d818420
Resource @ 0x8dfb9168 Shared 2 owning threads
Threads: 8e7413b8-01<*> 8d862630-01<*>
Resource @ 0x8dfb9128 Exclusively owned
Contention Count = 1
Threads: 8e7413b8-01<*>
Resource @ 0x8df86150 Shared 1 owning threads
Threads: 8dba25d8-01<*>
Resource @ 0x8df86110 Exclusively owned
Threads: 8e741b08-01<*>
Resource @ 0x8e3f17d0 Shared 1 owning threads
Threads: 8dc84db0-01<*>
Resource @ 0x8dc9c008 Exclusively owned
Threads: 8e72f480-01<*>
Resource @ 0x8dc9c1a8 Shared 1 owning threads
Threads: 8e72f480-01<*>
Resource @ 0x8e3ec7a0 Shared 1 owning threads
Threads: 8e7413b8-01<*>
Resource @ 0x8d85a218 Exclusively owned
Threads: 8d80fc70-01<*>
Resource @ 0x8e2d1988 Shared 1 owning threads
Threads: 8d7c5370-01<*>
Resource @ 0x8e3f1ac0 Shared 1 owning threads
Threads: 8e416458-01<*>
Resource @ 0x8dc11798 Exclusively owned
Contention Count = 1
Threads: 8e73f8d0-01<*>
Resource @ 0x8dc93298 Exclusively owned
Threads: 8e73f8d0-01<*>
Resource @ 0x8e2bb198 Shared 1 owning threads
Contention Count = 1
Threads: 8d80ac70-01<*>
Resource @ 0x8e06e6f8 Shared 1 owning threads
Threads: 8e72f480-01<*>
4346 total locks, 47 locks currently held
We don’t see High Contention (executive resources) pattern. What we actually see is a runaway system thread:
0: kd> !running
System Processors f (affinity mask)
Idle Processors d
Prcbs Current Next
1 f772f120 8e72f480 …………….
0: kd> !thread 8e72f480
THREAD 8e72f480 Cid 0004.00e4 Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 1
Not impersonating
DeviceMap d66018c0
Owning Process 8e7437a8 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 78418 Ticks: 919192 (0:03:59:22.375)
Context Switch Count 240
UserTime 00:00:00.000
KernelTime 03:59:29.203
Start Address 0×80848dbc
Stack Init b2f94000 Current b2f93584 Base b2f94000 Limit b2f91000 Call 0
Priority 17 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr Args to Child
[..]
b2f93a14 f7b4dae0 8d802108 d78220d0 b2f93be8 ModuleA!bar+0×2a0
b2f93c14 f7b527d0 8d802108 8e28d218 8e574860 ModuleA!foo+0×1e27
[…]
b2f93ddc 8088fa7e 80848dbc 00000000 00000000 nt!PspSystemThreadStartup+0×2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0×16
I highlighted in red this thread in the output of !locks command above. Many wait chains terminate at this thread (an example one is highlighted in blue above, 8d818870 -> 8d80fc70 -> 8dbe0388 -> 8e72f480). Stack trace collection shows ModuleA on top of stack traces of many threads (!stacks 0 ModuleA! filter command) but I don’t include its output here.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
