Manual dump, dynamic memory corruption, blocked threads, stack trace collection, multiple exceptions, wait chains and deadlock: pattern cooperation
CARE: Crash Analysis Report Environment
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
The following memory dump of a hanging process was manually generated:
Loading Dump File [Process.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER'
Applying default analysis command shows the following stack traces:
0:000> .kframes 100
Default stack trace depth is 0n256 frames
0:000> !analyze -v
[...]
STACK_TEXT:
009ef258 7c827d0b ntdll!KiFastSystemCallRet
009ef25c 7c83d236 ntdll!NtWaitForSingleObject+0xc
009ef298 7c83d281 ntdll!RtlpWaitOnCriticalSection+0×1a3
009ef2b8 7c82dabf ntdll!RtlEnterCriticalSection+0xa8
009ef358 7c82dab1 ntdll!LdrpGetProcedureAddress+0×128
009ef374 77e764ea ntdll!LdrGetProcedureAddress+0×18
009ef5d8 7c34c456 kernel32!UnhandledExceptionFilter+0×46f
009ef5f4 7c34957c msvcr71!_XcptFilter+0×15f
009ef600 7c34246e msvcr71!_endthreadex+0xb7
009ef628 7c828752 msvcr71!_except_handler3+0×61
009ef64c 7c828723 ntdll!ExecuteHandler2+0×26
009ef6f4 7c82855e ntdll!ExecuteHandler+0×24
009ef6f4 7c82be3e ntdll!KiUserExceptionDispatcher+0xe
009efa00 7c82a319 ntdll!RtlpFindEntry+0×68
009efc2c 7c3416b3 ntdll!RtlAllocateHeap+0×606
009efc6c 7c3416db msvcr71!_heap_alloc+0xe0
009efc74 7c360947 msvcr71!_nh_malloc+0×10
009efc80 0285f893 msvcr71!operator new+0xb
[…]
009effb8 77e64829 msvcr71!_endthreadex+0xa0
009effec 00000000 kernel32!BaseThreadStart+0×34
[...]
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT_STATUS_BREAKPOINT
We see the exeption was dispatched because of heap corruption and the unhandled exception filter is blocked waiting for a critical section. We can immediately recommend to enable full page heap. However, let’s explore the dump file further. By listing all threads we find that there were 2 exceptions with the second one having the following stack trace:
0:000> ~*kb
[...]
98 Id: be4.2ca4 Suspend: 1 Teb: 7ff68000 Unfrozen
ChildEBP RetAddr Args to Child
0f83c80c 7c827d0b 7c83d236 00000154 00000000 ntdll!KiFastSystemCallRet
0f83c810 7c83d236 00000154 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0f83c84c 7c83d281 00000154 00000004 0f83c8b0 ntdll!RtlpWaitOnCriticalSection+0×1a3
0f83c86c 7c82dabf 7c8877a0 00000000 00000000 ntdll!RtlEnterCriticalSection+0xa8
0f83c90c 7c82dab1 00000000 77e767cc 00000000 ntdll!LdrpGetProcedureAddress+0×128
0f83c928 77e764ea 00000000 77e767cc 00000000 ntdll!LdrGetProcedureAddress+0×18
0f83cb8c 77e792a3 0f83cbb4 77e61ac1 0f83cbbc kernel32!UnhandledExceptionFilter+0×46f
0f83cb94 77e61ac1 0f83cbbc 00000000 0f83cbbc kernel32!BaseThreadStart+0×4a
0f83cbbc 7c828752 0f83cca0 0f83ffdc 0f83ccbc kernel32!_except_handler3+0×61
0f83cbe0 7c828723 0f83cca0 0f83ffdc 0f83ccbc ntdll!ExecuteHandler2+0×26
0f83cc88 7c82855e 0f83c000 0f83ccbc 0f83cca0 ntdll!ExecuteHandler+0×24
0f83cc88 7c82be3e 0f83c000 0f83ccbc 0f83cca0 ntdll!KiUserExceptionDispatcher+0xe
0f83cf94 7c82a319 00340178 00000050 0b425f54 ntdll!RtlpFindEntry+0×68
0f83d1c0 7c3416b3 00340000 00000000 00000278 ntdll!RtlAllocateHeap+0×606
0f83d200 7c3416db 00000278 7c3416f8 00000278 msvcr71!_heap_alloc+0xe0
0f83d208 7c3416f8 00000278 00000000 003214fd msvcr71!_nh_malloc+0×10
0f83d214 003214fd 00000278 0f83d23c 023e5912 msvcr71!malloc+0xf
[…]
0f83f8b4 77ce33e1 0cf4ca00 0f83fa98 0000000c rpcrt4!Invoke+0×30
0f83fcb4 77ce35c4 00000000 00000000 0bf93d8c rpcrt4!NdrStubCall2+0×299
0f83fcd0 77c7ff7a 0bf93d8c 0beae6b8 0bf93d8c rpcrt4!NdrServerCall2+0×19
0f83fd04 77c8042d 0cf4f53e 0bf93d8c 0f83fdec rpcrt4!DispatchToStubInCNoAvrf+0×38
0f83fd58 77c80353 0000006f 00000000 0cf74358 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
0f83fd7c 77c7e0d4 0bf93d8c 00000000 0cf74358 rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
0f83fdbc 77c7e080 0bf93d8c 0bf93d44 00000000 rpcrt4!RPC_INTERFACE::DispatchToStubWithObject+0xc0
0f83fdfc 77c812f0 001c85c0 0bf0f510 00189d08 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×41e
0f83fe20 77c88678 0bf0f548 0f83fe38 001c85c0 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0f83ff84 77c88792 0f83ffac 77c8872d 0bf0f510 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0f83ff8c 77c8872d 0bf0f510 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
0f83ffac 77c7b110 00145520 0f83ffec 77e64829 rpcrt4!BaseCachedThreadRoutine+0×9d
0f83ffb8 77e64829 0be74ce0 00000000 00000000 rpcrt4!ThreadStartRoutine+0×1b
0f83ffec 00000000 77c7b0f5 0be74ce0 00000000 kernel32!BaseThreadStart+0×34
[...]
102 Id: be4.2ac0 Suspend: 1 Teb: 7ff4f000 Unfrozen
ChildEBP RetAddr Args to Child
134df91c 7c827d0b 7c83d236 000004e4 00000000 ntdll!KiFastSystemCallRet
134df920 7c83d236 000004e4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
134df95c 7c83d281 000004e4 00000004 00340000 ntdll!RtlpWaitOnCriticalSection+0×1a3
134df97c 7c82a264 00340608 00000000 0000008c ntdll!RtlEnterCriticalSection+0xa8
134dfba4 7c3423aa 00340000 00000008 0000008c ntdll!RtlAllocateHeap+0×313
134dfbe4 7c3422cb 00000001 0000008c 00000000 msvcr71!calloc+0xe6
134dfbfc 7c81a352 7c340000 00000002 00000000 msvcr71!_CRTDLL_INIT+0×138
134dfc1c 7c82ed97 7c34229f 7c340000 00000002 ntdll!LdrpCallInitRoutine+0×14
134dfcb8 7c82ec9f 134dfd28 134dfd28 00000000 ntdll!LdrpInitializeThread+0×10d
134dfd14 7c8284c5 134dfd28 7c800000 00000000 ntdll!_LdrpInitialize+0×16f
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0×25
If we inspect the list of locked critical sections and compare stack trace parameters for threads 2ac0 and 2ca4 we would see them deadlocked:
0:000> !locks
CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0
WaiterWoken No
LockCount 6
RecursionCount 1
OwningThread 2ac0
EntryCount 0
ContentionCount 36
*** Locked
CritSec +340608 at 00340608
WaiterWoken No
LockCount 32
RecursionCount 1
OwningThread 2ca4
EntryCount 0
ContentionCount 6f
*** Locked
[...]
Looking at other locks (omitted here) we can find additional wait chains but we employ default hang analysis command to find one chain for us:
0:000> !analyze -v -hang
[...]
LOADERLOCK_BLOCKED_API: UnhandledExceptionFilter:LdrGetProcedureAddress: LdrpGetProcedureAddress:
LOADERLOCK_OWNER_API: _LdrpInitialize:LdrpInitializeThread:LdrpCallInitRoutine:
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
2 be4.c20 Critical Section -->
x 98 be4.2ca4 Critical Section -->
x 102 be4.2ac0 Critical Section --^
WAIT_CHAIN_COMMAND: ~2s;k;;~98s;k;;~102s;k;;
BLOCKING_THREAD: 00002ca4
DEFAULT_BUCKET_ID: APPLICATION_HANG_DEADLOCK_HeapCorruption
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_DEADLOCK_HeapCorruption
[...]
- Dmitry Vostokov @ DumpAnalysis.org -
_1125.png)
Coming Soon:
Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
March 9th, 2009 at 4:57 pm
This looks similar to another deadlock I found with Firefox and Breakpad:
https://bugzilla.mozilla.org/show_bug.cgi?id=474254