Software Diagnostics Library

Crash Dump Analysis Patterns (Part 71)

Dynamic memory corruption patterns in user and kernel spaces are specializations of one big parent pattern called Corrupt Structure because crashes there happen due to corrupt or overwritten heap or pool control structures (for the latter see Double Free pattern). Another frequently seen specialization is called Critical Section Corruption which is the subject of this post. Critical sections are linked together through statically pre-allocated or heap-allocated helper structure (shown in magenta) although themselves they can be stored anywhere from static and stack area to heap:

0:001> dt -r1 ntdll!_RTL_CRITICAL_SECTION 77795240 +0×000 DebugInfo : 0×00175d28 _RTL_CRITICAL_SECTION_DEBUG +0×000 Type : 0 +0×002 CreatorBackTraceIndex : 0 +0×004 CriticalSection : 0×77795240 _RTL_CRITICAL_SECTION +0×008 ProcessLocksList : _LIST_ENTRY [ 0×173a08 - 0×173298 ] +0×010 EntryCount : 0 +0×014 ContentionCount : 0 +0×018 Spare : [2] 0 +0×004 LockCount : -1 +0×008 RecursionCount : 0 +0×00c OwningThread : (null) +0×010 LockSemaphore : (null) +0×014 SpinCount : 0

0:001> !address 77795240 77670000 : 77792000 - 00005000 Type 01000000 MEM_IMAGE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageImage FullPath C:\WINDOWS\system32\ole32.dll

0:001> !address 0×00175d28 00140000 : 00173000 - 0000d000 Type 00020000 MEM_PRIVATE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageHeap Handle 00140000

0:000> !locks

CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread 1184 EntryCount 0 ContentionCount b04707 *** Locked

0:000> dt -r1 _RTL_CRITICAL_SECTION 7c8877a0 +0×000 DebugInfo : 0×7c8877c0 _RTL_CRITICAL_SECTION_DEBUG +0×000 Type : 0 +0×002 CreatorBackTraceIndex : 0 +0×004 CriticalSection : 0×7c8877a0 _RTL_CRITICAL_SECTION +0×008 ProcessLocksList : _LIST_ENTRY [ 0×7c887be8 - 0×7c887bc8 ] +0×010 EntryCount : 0 +0×014 ContentionCount : 0xb04707 +0×018 Spare : [2] 0 +0×004 LockCount : -2 +0×008 RecursionCount : 1 +0×00c OwningThread : 0×00001184 +0×010 LockSemaphore : 0×0000013c +0×014 SpinCount : 0

0:000> !address 7c8877a0 7c800000 : 7c887000 - 00003000 Type 01000000 MEM_IMAGE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageImage FullPath C:\WINDOWS\system32\ntdll.dll

0:000> !address 0×7c8877c0 7c800000 : 7c887000 - 00003000 Type 01000000 MEM_IMAGE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageImage FullPath C:\WINDOWS\system32\ntdll.dll

Consider the case when CRITICAL_SECTION is defined on a stack and there was Local Buffer Overflow overwriting DebugInfo pointer. Then we have an example of Wild Pointer pattern and traversing the list of critical sections from this point will diverge into completely unrelated memory area or stop there. Consider another example of heap corruption or race condition overwriting ProcessLocksList or CriticalSection pointer. Then we have an instance of Wild Pointer pattern illustrated below:

0:000> !locks

CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread 1184 EntryCount 0 ContentionCount b04707 *** Locked

CritSec +1018de08 at 1018de08 WaiterWoken Yes LockCount -49153 RecursionCount 5046347 OwningThread 460050 EntryCount 0 ContentionCount 0 *** Locked

CritSec +1018ddd8 at 1018ddd8 WaiterWoken Yes LockCount -1 RecursionCount 0 OwningThread 0 *** Locked

CritSec +1018de28 at 1018de28 WaiterWoken Yes LockCount -1 RecursionCount 0 OwningThread 0 *** Locked

CritSec +1018de08 at 1018de08 WaiterWoken Yes LockCount -49153 RecursionCount 5046347 OwningThread 460050 EntryCount 0 ContentionCount 0 *** Locked

CritSec +1018de28 at 1018de28 WaiterWoken Yes LockCount -1 RecursionCount 0 OwningThread 0 *** Locked

CritSec +1018ddd8 at 1018ddd8 WaiterWoken Yes LockCount -1 RecursionCount 0 OwningThread 0 *** Locked

Scanned 841 critical sections

We see the signs of corruption at 1018de08 address which is interpreted as pointing to a locked critical section. To see where the corruption started we need to look at the list of all critical sections either locked or not locked:

0:000> !locks -v

CritSec ntdll!RtlCriticalSectionLock+0 at 7c887780 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 28

CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread 1184 EntryCount 0 ContentionCount b04707 *** Locked

CritSec ntdll!FastPebLock+0 at 7c887740 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 42c9

CritSec ntdll!RtlpCalloutEntryLock+0 at 7c888ea0 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec ntdll!PMCritSect+0 at 7c8883c0 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec ntdll!UMLogCritSect+0 at 7c888400 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec ntdll!RtlpProcessHeapsListLock+0 at 7c887960 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec +80608 at 00080608 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 22

[...]

CritSec cabinet!_adbgmsg+13c at 74fb4658 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec +c6c17c at 00c6c17c LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec +c6c0e4 at 00c6c0e4 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec at 1018de08 does not point back to the debug info at 00136a40 Perhaps the memory that held the critical section has been reused without calling DeleteCriticalSection() ?

CritSec +1018de08 at 1018de08 WaiterWoken Yes LockCount -49153 RecursionCount 5046347 OwningThread 460050 EntryCount 0 ContentionCount 0 *** Locked

CritSec at 1018ddd8 does not point back to the debug info at 00136a68 Perhaps the memory that held the critical section has been reused without calling DeleteCriticalSection() ?

CritSec +1018ddd8 at 1018ddd8 WaiterWoken Yes LockCount -1 RecursionCount 0 OwningThread 0 *** Locked

[...]

We see that the problem appears when the heap-allocated critical section at 00c6c0e4 address is linked to an inconsistent critical section at 0×1018de08 address where memory contains UNICODE string fragment:

0:000> !address 00c6c0e4 00c60000 : 00c60000 - 00010000 Type 00020000 MEM_PRIVATE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageHeap Handle 00c60000

0:000> dt -r1 _RTL_CRITICAL_SECTION 00c6c0e4 +0x000 DebugInfo : 0x00161140 _RTL_CRITICAL_SECTION_DEBUG +0x000 Type : 0 +0x002 CreatorBackTraceIndex : 0 +0x004 CriticalSection : 0x00c6c0e4 _RTL_CRITICAL_SECTION +0×008 ProcessLocksList : _LIST_ENTRY [ 0×136a48 - 0×119f58 ] +0×010 EntryCount : 0 +0×014 ContentionCount : 0 +0×018 Spare : [2] 0 +0×004 LockCount : -1 +0×008 RecursionCount : 0 +0×00c OwningThread : (null) +0×010 LockSemaphore : (null) +0×014 SpinCount : 0

0:000> dt -r _RTL_CRITICAL_SECTION_DEBUG 0×00136a48-0×008 +0×000 Type : 0 +0×002 CreatorBackTraceIndex : 0 +0×004 CriticalSection : 0×1018de08 _RTL_CRITICAL_SECTION +0×000 DebugInfo : 0×000d001b _RTL_CRITICAL_SECTION_DEBUG +0×000 Type : 0 +0×002 CreatorBackTraceIndex : 0 +0×004 CriticalSection : (null) +0×008 ProcessLocksList : _LIST_ENTRY [ 0×0 - 0×0 ] +0×010 EntryCount : 0 +0×014 ContentionCount : 0×37e3c700 +0×018 Spare : [2] 0×8000025 +0×004 LockCount : 196609 +0×008 RecursionCount : 5046347 +0×00c OwningThread : 0×00460050 +0×010 LockSemaphore : 0×00310033 +0×014 SpinCount : 0×520044 +0×008 ProcessLocksList : _LIST_ENTRY [ 0×136a70 - 0×161148 ] +0×000 Flink : 0×00136a70 _LIST_ENTRY [ 0×136a98 - 0×136a48 ] +0×000 Flink : 0×00136a98 _LIST_ENTRY [ 0×136ae8 - 0×136a70 ] +0×004 Blink : 0×00136a48 _LIST_ENTRY [ 0×136a70 - 0×161148 ] +0×004 Blink : 0×00161148 _LIST_ENTRY [ 0×136a48 - 0×119f58 ] +0×000 Flink : 0×00136a48 _LIST_ENTRY [ 0×136a70 - 0×161148 ] +0×004 Blink : 0×00119f58 _LIST_ENTRY [ 0×161148 - 0×16cc3c0 ] +0×010 EntryCount : 0 +0×014 ContentionCount : 0 +0×018 Spare : [2] 0×2e760000

0:000> !address 0x1018de08 10120000 : 10120000 - 00100000 Type 00020000 MEM_PRIVATE Protect 00000004 PAGE_READWRITE State 00001000 MEM_COMMIT Usage RegionUsageIsVAD

The address points miraculously to some DLL:

0:000> du 1018de08 1018de08 "....componentA.dll"

We might suggest that componentA.dll played some role there.

There are other messages from verbose version of !locks WinDbg command pointing to critical section problems:

CritSec componentB!Section+0 at 74004008 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

The CritSec componentC!Info+c at 72455074 has been RE-INITIALIZED. The critical section points to DebugInfo at 00107cc8 instead of 000f4788

CritSec componentC!Info+c at 72455074 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

CritSec componentD!foo+8ec0 at 0101add0 LockCount NOT LOCKED RecursionCount 0 OwningThread 0 EntryCount 0 ContentionCount 0

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Friday, July 11th, 2008 at 8:56 pm and is filed under Crash Dump Analysis, Crash Dump Patterns, Debugging. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

3 Responses to “Crash Dump Analysis Patterns (Part 71)”

Crash Dump Analysis » Blog Archive » Can computers debug? Says:
August 9th, 2008 at 9:35 pm
[…] It involves critical sections! Let’s see whether we have an instance of Critical Section Corruption pattern. The first disappointment comes when !locks command takes ages to finish so we break […]
Crash Dump Analysis » Blog Archive » The Mystery of Negative LockCount Says:
February 16th, 2009 at 4:27 pm
[…] Be aware though that negative values could also mean Critical Section Corruption pattern: […]
Crash Dump Analysis » Blog Archive » Strong process coupling, stack trace collection, critical section coruption and wait chains, message box, self-diagnosis and hidden exception and dynamic memory corruption: pattern cooperation Says:
April 26th, 2010 at 7:54 pm
[…] command shows wait chains and signs of critical section corruption. Here is the commented […]

You must be logged in to post a comment.

Crash Dump Analysis Patterns (Part 71)

3 Responses to “Crash Dump Analysis Patterns (Part 71)”

Leave a Reply