Crash Dump Analysis Patterns (Part 71)
Dynamic memory corruption patterns in user and kernel spaces are specializations of one big parent pattern called Corrupt Structure because crashes there happen due to corrupt or overwritten heap or pool control structures (for the latter see Double Free pattern). Another frequently seen specialization is called Critical Section Corruption which is the subject of this post. Critical sections are linked together through statically pre-allocated or heap-allocated helper structure (shown in magenta) although themselves they can be stored anywhere from static and stack area to heap:
0:001> dt -r1 ntdll!_RTL_CRITICAL_SECTION 77795240
+0×000 DebugInfo : 0×00175d28 _RTL_CRITICAL_SECTION_DEBUG
+0×000 Type : 0
+0×002 CreatorBackTraceIndex : 0
+0×004 CriticalSection : 0×77795240 _RTL_CRITICAL_SECTION
+0×008 ProcessLocksList : _LIST_ENTRY [ 0×173a08 - 0×173298 ]
+0×010 EntryCount : 0
+0×014 ContentionCount : 0
+0×018 Spare : [2] 0
+0×004 LockCount : -1
+0×008 RecursionCount : 0
+0×00c OwningThread : (null)
+0×010 LockSemaphore : (null)
+0×014 SpinCount : 0
0:001> !address 77795240
77670000 : 77792000 - 00005000
Type 01000000 MEM_IMAGE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINDOWS\system32\ole32.dll
0:001> !address 0×00175d28
00140000 : 00173000 - 0000d000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageHeap
Handle 00140000
0:000> !locks
CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread 1184
EntryCount 0
ContentionCount b04707
*** Locked
0:000> dt -r1 _RTL_CRITICAL_SECTION 7c8877a0
+0×000 DebugInfo : 0×7c8877c0 _RTL_CRITICAL_SECTION_DEBUG
+0×000 Type : 0
+0×002 CreatorBackTraceIndex : 0
+0×004 CriticalSection : 0×7c8877a0 _RTL_CRITICAL_SECTION
+0×008 ProcessLocksList : _LIST_ENTRY [ 0×7c887be8 - 0×7c887bc8 ]
+0×010 EntryCount : 0
+0×014 ContentionCount : 0xb04707
+0×018 Spare : [2] 0
+0×004 LockCount : -2
+0×008 RecursionCount : 1
+0×00c OwningThread : 0×00001184
+0×010 LockSemaphore : 0×0000013c
+0×014 SpinCount : 0
0:000> !address 7c8877a0
7c800000 : 7c887000 - 00003000
Type 01000000 MEM_IMAGE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINDOWS\system32\ntdll.dll
0:000> !address 0×7c8877c0
7c800000 : 7c887000 - 00003000
Type 01000000 MEM_IMAGE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINDOWS\system32\ntdll.dll
Consider the case when CRITICAL_SECTION is defined on a stack and there was Local Buffer Overflow overwriting DebugInfo pointer. Then we have an example of Wild Pointer pattern and traversing the list of critical sections from this point will diverge into completely unrelated memory area or stop there. Consider another example of heap corruption or race condition overwriting ProcessLocksList or CriticalSection pointer. Then we have an instance of Wild Pointer pattern illustrated below:
0:000> !locks
CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread 1184
EntryCount 0
ContentionCount b04707
*** Locked
CritSec +1018de08 at 1018de08
WaiterWoken Yes
LockCount -49153
RecursionCount 5046347
OwningThread 460050
EntryCount 0
ContentionCount 0
*** Locked
CritSec +1018ddd8 at 1018ddd8
WaiterWoken Yes
LockCount -1
RecursionCount 0
OwningThread 0
*** Locked
CritSec +1018de28 at 1018de28
WaiterWoken Yes
LockCount -1
RecursionCount 0
OwningThread 0
*** Locked
CritSec +1018de08 at 1018de08
WaiterWoken Yes
LockCount -49153
RecursionCount 5046347
OwningThread 460050
EntryCount 0
ContentionCount 0
*** Locked
CritSec +1018de28 at 1018de28
WaiterWoken Yes
LockCount -1
RecursionCount 0
OwningThread 0
*** Locked
CritSec +1018ddd8 at 1018ddd8
WaiterWoken Yes
LockCount -1
RecursionCount 0
OwningThread 0
*** Locked
Scanned 841 critical sections
We see the signs of corruption at 1018de08 address which is interpreted as pointing to a locked critical section. To see where the corruption started we need to look at the list of all critical sections either locked or not locked:
0:000> !locks -v
CritSec ntdll!RtlCriticalSectionLock+0 at 7c887780
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 28
CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread 1184
EntryCount 0
ContentionCount b04707
*** Locked
CritSec ntdll!FastPebLock+0 at 7c887740
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 42c9
CritSec ntdll!RtlpCalloutEntryLock+0 at 7c888ea0
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec ntdll!PMCritSect+0 at 7c8883c0
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec ntdll!UMLogCritSect+0 at 7c888400
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec ntdll!RtlpProcessHeapsListLock+0 at 7c887960
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec +80608 at 00080608
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 22
[...]
CritSec cabinet!_adbgmsg+13c at 74fb4658
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec +c6c17c at 00c6c17c
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec +c6c0e4 at 00c6c0e4
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec at 1018de08 does not point back to the debug info at 00136a40
Perhaps the memory that held the critical section has been reused without calling DeleteCriticalSection() ?
CritSec +1018de08 at 1018de08
WaiterWoken Yes
LockCount -49153
RecursionCount 5046347
OwningThread 460050
EntryCount 0
ContentionCount 0
*** Locked
CritSec at 1018ddd8 does not point back to the debug info at 00136a68
Perhaps the memory that held the critical section has been reused without calling DeleteCriticalSection() ?
CritSec +1018ddd8 at 1018ddd8
WaiterWoken Yes
LockCount -1
RecursionCount 0
OwningThread 0
*** Locked
[...]
We see that the problem appears when the heap-allocated critical section at 00c6c0e4 address is linked to an inconsistent critical section at 0×1018de08 address where memory contains UNICODE string fragment:
0:000> !address 00c6c0e4
00c60000 : 00c60000 - 00010000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageHeap
Handle 00c60000
0:000> dt -r1 _RTL_CRITICAL_SECTION 00c6c0e4
+0x000 DebugInfo : 0x00161140 _RTL_CRITICAL_SECTION_DEBUG
+0x000 Type : 0
+0x002 CreatorBackTraceIndex : 0
+0x004 CriticalSection : 0x00c6c0e4 _RTL_CRITICAL_SECTION
+0×008 ProcessLocksList : _LIST_ENTRY [ 0×136a48 - 0×119f58 ]
+0×010 EntryCount : 0
+0×014 ContentionCount : 0
+0×018 Spare : [2] 0
+0×004 LockCount : -1
+0×008 RecursionCount : 0
+0×00c OwningThread : (null)
+0×010 LockSemaphore : (null)
+0×014 SpinCount : 0
0:000> dt -r _RTL_CRITICAL_SECTION_DEBUG 0×00136a48-0×008
+0×000 Type : 0
+0×002 CreatorBackTraceIndex : 0
+0×004 CriticalSection : 0×1018de08 _RTL_CRITICAL_SECTION
+0×000 DebugInfo : 0×000d001b _RTL_CRITICAL_SECTION_DEBUG
+0×000 Type : 0
+0×002 CreatorBackTraceIndex : 0
+0×004 CriticalSection : (null)
+0×008 ProcessLocksList : _LIST_ENTRY [ 0×0 - 0×0 ]
+0×010 EntryCount : 0
+0×014 ContentionCount : 0×37e3c700
+0×018 Spare : [2] 0×8000025
+0×004 LockCount : 196609
+0×008 RecursionCount : 5046347
+0×00c OwningThread : 0×00460050
+0×010 LockSemaphore : 0×00310033
+0×014 SpinCount : 0×520044
+0×008 ProcessLocksList : _LIST_ENTRY [ 0×136a70 - 0×161148 ]
+0×000 Flink : 0×00136a70 _LIST_ENTRY [ 0×136a98 - 0×136a48 ]
+0×000 Flink : 0×00136a98 _LIST_ENTRY [ 0×136ae8 - 0×136a70 ]
+0×004 Blink : 0×00136a48 _LIST_ENTRY [ 0×136a70 - 0×161148 ]
+0×004 Blink : 0×00161148 _LIST_ENTRY [ 0×136a48 - 0×119f58 ]
+0×000 Flink : 0×00136a48 _LIST_ENTRY [ 0×136a70 - 0×161148 ]
+0×004 Blink : 0×00119f58 _LIST_ENTRY [ 0×161148 - 0×16cc3c0 ]
+0×010 EntryCount : 0
+0×014 ContentionCount : 0
+0×018 Spare : [2] 0×2e760000
0:000> !address 0x1018de08
10120000 : 10120000 - 00100000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
The address points miraculously to some DLL:
0:000> du 1018de08
1018de08 "....componentA.dll"
We might suggest that componentA.dll played some role there.
There are other messages from verbose version of !locks WinDbg command pointing to critical section problems:
CritSec componentB!Section+0 at 74004008
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
The CritSec componentC!Info+c at 72455074 has been RE-INITIALIZED.
The critical section points to DebugInfo at 00107cc8 instead of 000f4788
CritSec componentC!Info+c at 72455074
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
CritSec componentD!foo+8ec0 at 0101add0
LockCount NOT LOCKED
RecursionCount 0
OwningThread 0
EntryCount 0
ContentionCount 0
- Dmitry Vostokov @ DumpAnalysis.org -
August 9th, 2008 at 9:35 pm
[…] It involves critical sections! Let’s see whether we have an instance of Critical Section Corruption pattern. The first disappointment comes when !locks command takes ages to finish so we break […]
February 16th, 2009 at 4:27 pm
[…] Be aware though that negative values could also mean Critical Section Corruption pattern: […]
April 26th, 2010 at 7:54 pm
[…] command shows wait chains and signs of critical section corruption. Here is the commented […]