Crash Dump Analysis Patterns (Part 29b)
Previously we discussed High Contention pattern in kernel mode involving executive resources. The same pattern can be observed in user space involving critical sections guarding shared regions like serialized process heap or a memory database, for example, in one Windows service process during increased workload:
0:000> !locks
CritSec +310608 at 00310608
WaiterWoken No
LockCount 6
RecursionCount 1
OwningThread d9c
EntryCount 0
ContentionCount 453093
*** Locked
CritSec +8f60f78 at 08f60f78
WaiterWoken No
LockCount 8
RecursionCount 1
OwningThread d9c
EntryCount 0
ContentionCount af7f0
*** Locked
CritSec +53bf8f10 at 53bf8f10
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread 1a9c
EntryCount 0
ContentionCount e
*** Locked
Scanned 7099 critical sections
Looking at the owning thread we see that contention involves process heap:
0:000> ~~[d9c]kL
ChildEBP RetAddr
0e2ff9d4 7c81e845 ntdll!RtlpFindAndCommitPages+0×14e
0e2ffa0c 7c81e4ef ntdll!RtlpExtendHeap+0xa6
0e2ffc38 7c3416b3 ntdll!RtlAllocateHeap+0×645
0e2ffc78 7c3416db msvcr71!_heap_alloc+0xe0
0e2ffc80 7c3416f8 msvcr71!_nh_malloc+0×10
0e2ffc8c 672e14fd msvcr71!malloc+0xf
0e2ffc98 0040bc28 dll!MemAlloc+0xd
[…]
0e2fff84 7c349565 dll!WorkItemThread+0×152
0e2fffb8 77e6608b msvcr71!_endthreadex+0xa0
0e2fffec 00000000 kernel32!BaseThreadStart+0×34
However two critical section addresses belong to the same heap:
0:000> !address 00310608
00310000 : 00310000 - 00010000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageHeap
Handle 00310000
0:000> !address 08f60f78
08f30000 : 08f30000 - 00200000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageHeap
Handle 00310000
Lock contention is confirmed in heap statistics as well:
0:000> !heap -s
LFH Key : 0x07262959
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
00140000 00000002 8192 2876 3664 631 140 46 0 1e L
External fragmentation 21 % (140 free blocks)
00240000 00008000 64 12 12 10 1 1 0 0
Virtual block: 0ea20000 - 0ea20000 (size 00000000)
Virtual block: 0fa30000 - 0fa30000 (size 00000000)
00310000 00001002 1255320 961480 1249548 105378 0 16830 2 453093 L
Virtual address fragmentation 23 % (16830 uncommited ranges)
Lock contention 4534419
003f0000 00001002 64 36 36 0 0 1 0 0 L
00610000 00001002 64 16 16 4 2 1 0 0 L
[…]
- Dmitry Vostokov @ DumpAnalysis.org -
October 9th, 2009 at 2:38 pm
[…] section it owns shows high contention count […]
October 19th, 2009 at 9:10 pm
[…] Examining critical section locks showed this section to be the only one locked and having high contention: […]
June 21st, 2010 at 11:19 am
[…] we introduce an icon for High Contention (critical sections) […]
September 21st, 2010 at 10:47 am
[…] High Contention (critical sections) […]