Crash Dump Analysis

CARE: Crash Analysis Report Environment

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

Previously we discussed High Contention pattern in kernel mode involving executive resources. The same pattern can be observed in user space involving critical sections guarding shared regions like serialized process heap or a memory database, for example, in one Windows service process during increased workload:

0:000> !locks

CritSec +310608 at 00310608
WaiterWoken        No
LockCount          6
RecursionCount     1
OwningThread       d9c
EntryCount         0
ContentionCount    453093
*** Locked

CritSec +8f60f78 at 08f60f78
WaiterWoken        No
LockCount          8
RecursionCount     1
OwningThread       d9c
EntryCount         0
ContentionCount    af7f0
*** Locked

CritSec +53bf8f10 at 53bf8f10
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       1a9c
EntryCount         0
ContentionCount    e
*** Locked

Scanned 7099 critical sections

Looking at the owning thread we see that contention involves process heap: 

0:000> ~~[d9c]kL
ChildEBP RetAddr 
0e2ff9d4 7c81e845 ntdll!RtlpFindAndCommitPages+0×14e
0e2ffa0c 7c81e4ef ntdll!RtlpExtendHeap+0xa6
0e2ffc38 7c3416b3 ntdll!RtlAllocateHeap+0×645
0e2ffc78 7c3416db msvcr71!_heap_alloc+0xe0
0e2ffc80 7c3416f8 msvcr71!_nh_malloc+0×10
0e2ffc8c 672e14fd msvcr71!malloc+0xf
0e2ffc98 0040bc28 dll!MemAlloc+0xd
[…]
0e2fff84 7c349565 dll!WorkItemThread+0×152
0e2fffb8 77e6608b msvcr71!_endthreadex+0xa0
0e2fffec 00000000 kernel32!BaseThreadStart+0×34

However two critical section addresses belong to the same heap: 

0:000> !address 00310608
    00310000 : 00310000 - 00010000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   00310000

0:000> !address 08f60f78
    08f30000 : 08f30000 - 00200000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   00310000

Lock contention is confirmed in heap statistics as well: 

0:000> !heap -s
LFH Key                   : 0x07262959
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                    (k)     (k)    (k)     (k) length      blocks cont. heap
00140000 00000002    8192   2876   3664    631   140    46    0     1e   L 
    External fragmentation  21 % (140 free blocks)
00240000 00008000      64     12     12     10     1     1    0      0     
Virtual block: 0ea20000 - 0ea20000 (size 00000000)
Virtual block: 0fa30000 - 0fa30000 (size 00000000)
00310000 00001002 1255320 961480 1249548 105378     0 16830    2 453093   L 
    Virtual address fragmentation  23 % (16830 uncommited ranges)
    Lock contention  4534419
003f0000 00001002      64     36     36      0     0     1    0      0   L 
00610000 00001002      64     16     16      4     2     1    0      0   L
[…]

- Dmitry Vostokov @ DumpAnalysis.org -

           

Announcements

Coming Soon:

Debugging Notebook: Essential Concepts, WinDbg Commands and Tools

Crash Dump Analysis for System Administrators and Support Engineers

New Magazines:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers

New Books:

Memory Dump Analysis Anthology, Volume 3

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

x64 Windows Debugging: Practical Foundations

Also available:

Windows Debugging: Practical Foundations

DLL List Landscape: The Art from Computer Memory Space

Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

WinDbg: A Reference Poster and Learning Cards

Memory Dump Analysis Anthology, Volume 2

Memory Dump Analysis Anthology, Volume 1

New Children's Book:

Baby Turing

This entry was posted on Wednesday, September 10th, 2008 at 2:46 pm and is filed under Crash Dump Analysis, Crash Dump Patterns. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.