Software Diagnostics Library

Now I resume previously introduced memory analysis succession patterns post series. In this part I introduce a case study where Wait Chains (executive resources) and Swarm of Shared Locks were probably resulted from a Spiking Thread. We have these resource locks:

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****

Resource @ 0x808a6860    Shared 9 owning threads
     Threads: 8e739b40-01<*> 8e07db58-01<*> 8e455d18-01<*> 8df72958-01<*>
              8e356620-01<*> 8d7e9700-01<*> 8e73a660-01<*> 8dbb0af0-01<*>
              8e01c9b8-01<*>

Resource @ 0xf7b6d5b0    Shared 2 owning threads
    Contention Count = 1
     Threads: 8e741898-01<*> 8e73fb40-01<*>

Resource @ 0x8e5a1534    Shared 12 owning threads
    Contention Count = 76
    NumberOfSharedWaiters = 7
    NumberOfExclusiveWaiters = 1
     Threads: 8e73fb40-01    8d80fc70-01<*> 8dc226c0-01<*> 8dc84db0-01<*>
              8e416458-01<*> 8dbf1630-01<*> 8e740db0-01<*> 8d7e9700-01<*>
              8d818420-01<*> 8e7413b8-01<*> 8e739020-01    8d80ac70-01<*>
              8e30ec88-01<*> 8e7408d0-01    8dd022a8-01    8e59a2f8-01   
              8dc4d300-01    8dc36278-01<*> 8e060078-01   
     Threads Waiting On Exclusive Access:
              8d818870

Resource @ 0x8e5a1368    Shared 3 owning threads
    Contention Count = 132
    NumberOfExclusiveWaiters = 9
     Threads: 8d7c5370-01<*> 8e416458-01<*> 8e72f480-01<*>
     Threads Waiting On Exclusive Access:
              8d7e9700       8d80ac70       8e741b08       8dc84db0      
              8df72958       8e73e8d0       8dbe0388       8e7413b8      
              8e741898      

Resource @ 0x8e74a3b0    Exclusively owned
    Contention Count = 11
    NumberOfSharedWaiters = 2
    NumberOfExclusiveWaiters = 1
     Threads: 8dbe0388-01<*> 8e73e660-01    8e740020-01   
     Threads Waiting On Exclusive Access:
              8d80fc70      

Resource @ 0x8e54f810    Exclusively owned
    Contention Count = 118
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 2
     Threads: 8e72f480-01<*> 8e73f8d0-01   
     Threads Waiting On Exclusive Access:
              8d7c5370       8e416458      

Resource @ 0x8e6db008    Shared 1 owning threads
     Threads: 8e73f8d0-01<*>

Resource @ 0x8e75e3c0    Exclusively owned
     Threads: 8d7e9700-01<*>

Resource @ 0x8e6f14b0    Exclusively owned
    Contention Count = 7
    NumberOfSharedWaiters = 2
     Threads: 8d80fc70-01<*> 8dbf1630-01    8e73f3f0-01   

Resource @ 0x8e707618    Exclusively owned
     Threads: 8dc84db0-01<*>

Resource @ 0x8e6c1780    Shared 1 owning threads
    Contention Count = 3
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 1
     Threads: 8e741b08-01<*> 8e73fdb0-01   
     Threads Waiting On Exclusive Access:
              8dc36278      

Resource @ 0x8e1fa370    Exclusively owned
     Threads: 8df72958-01<*>

Resource @ 0x8e290b38    Exclusively owned
     Threads: 8df72958-01<*>

Resource @ 0x8e692be0    Shared 1 owning threads
    Contention Count = 12
     Threads: 8df72958-01<*>

Resource @ 0x8e3a03e0    Exclusively owned
    Contention Count = 4
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 1
     Threads: 8dbe0388-01<*> 8e740660-01   
     Threads Waiting On Exclusive Access:
              8e30ec88      

Resource @ 0x8e33d6a8    Exclusively owned
    Contention Count = 1
     Threads: 8dc36278-01<*>

Resource @ 0x8e33d640    Exclusively owned
    Contention Count = 1
     Threads: 8dc36278-01<*>

Resource @ 0x8e17f890    Exclusively owned
    Contention Count = 2
    NumberOfExclusiveWaiters = 1
     Threads: 8dc84db0-01<*>
     Threads Waiting On Exclusive Access:
              8e740db0      

Resource @ 0x8e17f828    Exclusively owned
     Threads: 8dc84db0-01<*>

Resource @ 0x8e09fc40    Exclusively owned
    Contention Count = 3
    NumberOfSharedWaiters = 2
     Threads: 8e416458-01<*> 8e076770-01    8dbf5b70-01   

Resource @ 0x8e09fbd8    Exclusively owned
     Threads: 8e416458-01<*>

Resource @ 0x8df021a0    Exclusively owned
     Threads: 8d7e9700-01<*>

Resource @ 0x8dffce50    Exclusively owned
    Contention Count = 2
    NumberOfExclusiveWaiters = 1
     Threads: 8dbf1630-01<*>
     Threads Waiting On Exclusive Access:
              8dc226c0      

Resource @ 0x8df147f8    Exclusively owned
    Contention Count = 4
    NumberOfSharedWaiters = 1
     Threads: 8dbf1630-01<*> 8e7403f0-01   

Resource @ 0x8e599de8    Exclusively owned
     Threads: 8d7c5370-01<*>

Resource @ 0x8e324ee8    Shared 1 owning threads
    Contention Count = 1
     Threads: 8d7c5370-01<*>

Resource @ 0x8e21bc60    Exclusively owned
     Threads: 8dbe0388-01<*>

Resource @ 0x8e446f18    Exclusively owned
     Threads: 8d80fc70-01<*>

Resource @ 0x8df65570    Shared 1 owning threads
    Contention Count = 3
     Threads: 8dc5f698-01<*>

Resource @ 0x8e085100    Exclusively owned
    Contention Count = 1
     Threads: 8e73e8d0-01<*>

Resource @ 0x8dbe51b8    Exclusively owned
     Threads: 8dbe0388-01<*>

Resource @ 0x8df458c8    Exclusively owned
    Contention Count = 1
    NumberOfExclusiveWaiters = 1
     Threads: 8d80ac70-01<*>
     Threads Waiting On Exclusive Access:
              8d818420      

Resource @ 0x8dfb9168    Shared 2 owning threads
     Threads: 8e7413b8-01<*> 8d862630-01<*>

Resource @ 0x8dfb9128    Exclusively owned
    Contention Count = 1
     Threads: 8e7413b8-01<*>

Resource @ 0x8df86150    Shared 1 owning threads
     Threads: 8dba25d8-01<*>

Resource @ 0x8df86110    Exclusively owned
     Threads: 8e741b08-01<*>

Resource @ 0x8e3f17d0    Shared 1 owning threads
     Threads: 8dc84db0-01<*>

Resource @ 0x8dc9c008    Exclusively owned
     Threads: 8e72f480-01<*>

Resource @ 0x8dc9c1a8    Shared 1 owning threads
     Threads: 8e72f480-01<*>

Resource @ 0x8e3ec7a0    Shared 1 owning threads
     Threads: 8e7413b8-01<*>

Resource @ 0x8d85a218    Exclusively owned
     Threads: 8d80fc70-01<*>

Resource @ 0x8e2d1988    Shared 1 owning threads
     Threads: 8d7c5370-01<*>

Resource @ 0x8e3f1ac0    Shared 1 owning threads
     Threads: 8e416458-01<*>

Resource @ 0x8dc11798    Exclusively owned
    Contention Count = 1
     Threads: 8e73f8d0-01<*>

Resource @ 0x8dc93298    Exclusively owned
     Threads: 8e73f8d0-01<*>

Resource @ 0x8e2bb198    Shared 1 owning threads
    Contention Count = 1
     Threads: 8d80ac70-01<*>

Resource @ 0x8e06e6f8    Shared 1 owning threads
     Threads: 8e72f480-01<*>

4346 total locks, 47 locks currently held

We don’t see High Contention (executive resources) pattern. What we actually see is a runaway system thread:

0: kd> !running

System Processors f (affinity mask)
  Idle Processors d

Prcbs  Current   Next   
  1    f772f120  8e72f480            …………….

0: kd> !thread 8e72f480
THREAD 8e72f480  Cid 0004.00e4  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 1
Not impersonating
DeviceMap                 d66018c0
Owning Process            8e7437a8       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      78418          Ticks: 919192 (0:03:59:22.375)
Context Switch Count      240            
UserTime                  00:00:00.000
KernelTime                03:59:29.203
Start Address 0×80848dbc
Stack Init b2f94000 Current b2f93584 Base b2f94000 Limit b2f91000 Call 0
Priority 17 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
[..]
b2f93a14 f7b4dae0 8d802108 d78220d0 b2f93be8 ModuleA!bar+0×2a0
b2f93c14 f7b527d0 8d802108 8e28d218 8e574860 ModuleA!foo+0×1e27
[…]
b2f93ddc 8088fa7e 80848dbc 00000000 00000000 nt!PspSystemThreadStartup+0×2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0×16

I highlighted in red this thread in the output of !locks command above. Many wait chains terminate at this thread (an example one is highlighted in blue above, 8d818870 -> 8d80fc70 -> 8dbe0388 -> 8e72f480). Stack trace collection shows ModuleA on top of stack traces of many threads (!stacks 0 ModuleA! filter command) but I don’t include its output here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This entry was posted on Thursday, June 3rd, 2010 at 11:10 pm and is filed under Crash Dump Analysis, Crash Dump Patterns, Debugging. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.