Crash Dump Analysis Patterns (Part 9a)
Next pattern is Deadlock. If you don’t know what “deadlock” is read Dumps for Dummes (Part 4). Deadlocks do not only happen with synchronization primitives like mutexes, events or more complex objects (built upon primitives) like critical sections or executive resources (ERESOURCE). They can happen from high level or systems perspective in inter-process or inter-component communication, for example, mutually waiting on messages: GUI window messages, LPC messages, RPC calls. This is a big pattern and I’m going to split it into several parts.
How can we see deadlocks in dumps? Let’s start with user dumps and critical sections.
First I would recommend to read the following excellent MSDN article to understand various members of CRITICAL_SECTION structure:
Break Free of Code Deadlocks in Critical Sections Under Windows
WinDbg !locks command will examine process critical section list and display all locked critical sections, lock count and thread id of current critical section owner. This is the output from a dump of hanging Windows print spooler process (spoolsv.exe):
0:000> !locks
CritSec NTDLL!LoaderLock+0 at 784B0348
LockCount 4
RecursionCount 1
OwningThread 624
EntryCount 6c3
ContentionCount 6c3
*** Locked
CritSec LOCALSPL!SpoolerSection+0 at 76AB8070
LockCount 3
RecursionCount 1
OwningThread 1c48
EntryCount 646
ContentionCount 646
*** Locked
If we look at threads #624 and #1c48 we could see them mutually waiting for each other:
-
TID#624 owns CritSec 784B0348 and is waiting for CritSec 76AB8070
-
TID#1c48 owns CritSec 76AB8070 and is waiting for CritSec 784B0348
0:000>~*kv
. 12 Id: bc0.624 Suspend: 1 Teb: 7ffd3000 Unfrozen
0000024c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb
76ab8000 76a815ef 76ab8070 NTDLL!RtlpWaitForCriticalSection+0×9e
76ab8070 76a844f8 00cd1f38 NTDLL!RtlEnterCriticalSection+0×46
00cd1f38 76a8a1d7 00000000 LOCALSPL!EnterSplSem+0xb
00000000 00000000 00cd1f38 LOCALSPL!FindSpoolerByNameIncRef+0×1f
00000000 777f19bc 00000001 LOCALSPL!LocalGetPrinterDriverDirectory+0xe
00000000 777f19bc 00000001 spoolss!GetPrinterDriverDirectoryW+0×59
00000000 777f19bc 00000001 spoolsv!YGetPrinterDriverDirectory+0×27
00000000 777f19bc 00000001 WINSPOOL!GetPrinterDriverDirectoryW+0×7b
50000000 00000001 00000000 BRHLUI04+0×14ea
50002ea0 50000000 00000001 BRHLUI04!DllGetClassObject+0×1705
00000000 00000000 000cb570 NTDLL!LdrpRunInitializeRoutines+0×1df
000cc8f8 0288ea30 0288ea38 NTDLL!LdrpLoadDll+0×2e6
000cc8f8 0288ea30 0288ea38 NTDLL!LdrLoadDll+0×17)
000c1258 00000000 00000008 KERNEL32!LoadLibraryExW+0×231
000c150c 0288efd8 00000000 UNIDRVUI!PLoadCommonInfo+0×17e
000c150c 0288efd8 00000007 UNIDRVUI!DwDeviceCapabilities+0×1a
00070000 00071378 00000045 UNIDRVUI!DrvDeviceCapabilities+0×19
. 13 Id: bc0.1c48 Suspend: 1 Teb: 7ffd2000 Unfrozen
0000010c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb
784b0301 78468d38 784b0348 NTDLL!RtlpWaitForCriticalSection+0×9e
784b0348 74fb4344 00000000 NTDLL!RtlEnterCriticalSection+0×46
74fb0000 02c0f2a8 00000000 NTDLL!LdrpGetProcedureAddress+0×122
74fb0000 02c0f2a8 00000000 NTDLL!LdrGetProcedureAddress+0×17
74fb0000 74fb4344 02c0f449 KERNEL32!GetProcAddress+0×41
017924b0 00000000 00000001 ws2_32!CheckForHookersOrChainers+0×1f
00000101 02c0f344 017924b0 ws2_32!WSAStartup+0×10f
00cdf79c 02c0f4f4 76a8c9bc LOCALSPL!GetDNSMachineName+0×1e
00000000 76a8c9bc 780276a2 LOCALSPL!GetPrinterUrl+0×2c
0176f570 ffffffff 01000000 LOCALSPL!UpdateDsSpoolerKey+0×322
0176f570 76a8c9bc 01792b90 LOCALSPL!RecreateDsKey+0×50
00000000 00000002 01792b90 LOCALSPL!SplAddPrinter+0×521
01791faa 0176a684 76a5cd34 WIN32SPL!InternalAddPrinterConnection+0×1b4
01791faa 02c0fa00 02c0fabc WIN32SPL!AddPrinterConnectionW+0×15
00076f1c 02c0fabc 01006873 spoolss!AddPrinterConnectionW+0×49
00076f1c 00000001 77107fb0 spoolsv!YAddPrinterConnection+0×17
00076f1c 02020202 00000001 spoolsv!RpcAddPrinterConnection+0xb
01006868 02c0fac0 00000001 rpcrt4!Invoke+0×30
00000000 00000000 000d22c8 rpcrt4!NdrStubCall2+0×655
000d22c8 00076fe0 000d22c8 rpcrt4!NdrServerCall2+0×17
010045fc 000d22c8 02c0fe0c rpcrt4!DispatchToStubInC+0×32
0000002b 00000000 02c0fe0c rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×100
000d22c8 00000000 02c0fe0c rpcrt4!RPC_INTERFACE::DispatchToStub+0×5e
000d3210 00076608 813b0013 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×1dd
000d21d0 02c0fe50 000d3210 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×10c
770c9ad0 00076608 770cb6d8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×229
00076608 770cb6d8 0288f9a8 rpcrt4!RecvLotsaCallsWrapper+0×9
00074a50 02c0ffec 77e7438b rpcrt4!BaseCachedThreadRoutine+0×11f
00076e68 770cb6d8 0288f9a8 rpcrt4!ThreadStartRoutine+0×18
770d1c54 00076e68 00000000 KERNEL32!BaseThreadStart+0×52
This analysis looks pretty simple and easy. What about kernel and complete memory dumps? Of course we cannot see user space critical sections in kernel memory dumps but we can see them in complete memory dumps after switching to appropriate process context and using !ntsdexts.locks. This can be done via simple script adapted from debugger.chm: Deadlocks and Critical Sections
Why it is so easy to see deadlocks when critical sections are involved? Because their structures have a member that records their owner. So it is very easy to map them to corresponding threads. The same is with kernel ERESOURCE synchronization objects (we will see them in the next part). Other objects do not have an owner, for example, in case of events it is not so easy to find an owner just by looking at an event object. You need to examine thread call stacks, other structures or have access to source code.
- Dmitry Vostokov @ DumpAnalysis.org -
May 29th, 2007 at 6:14 am
Hi Dmitry,
First i would like to appreciate your effort, i never ever seen such a dump analysis web site. Please let me know how can i share my thought also.
Currently i am facing one problem in locking thread. You have taken example of two thread locking, but if three or more thread getting lock then how we can identify which thread waiting for which critical section.
Ex.-
0:005> !locks
CritSec netiqms!__onexitbegin+50cbc at 006f1da8
WaiterWoken No
LockCount 49
RecursionCount 1
OwningThread 19a4
EntryCount 0
ContentionCount 2a51
*** Locked
CritSec netiqms!__onexitbegin+5131c at 006f2408
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread 10f4
EntryCount 0
ContentionCount 0
*** Locked
CritSec +6e8827c at 06e8827c
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread 19a4
EntryCount 0
ContentionCount 0
*** Locked
Scanned 2404 critical sections
Please let me know how can i explain above thread locking.
please suggest me if any book or help available to figure out or analyze dump.
Thanks in advance.
Regards,
Krishna
May 29th, 2007 at 10:43 am
Hi Krishna,
In the example you provided you have only one critical section which blocks other threads:
BlockedThreads = LockCount - (RecursionCount - 1) = 49 - (1 - 1)
CritSec netiqms!__onexitbegin+50cbc at 006f1da8
WaiterWoken No
LockCount 49
RecursionCount 1
OwningThread 19a4
EntryCount 0
ContentionCount 2a51
*** Locked
So here you can examine thread’s call stack 19a4 by looking at ~*kv output and find other waiting threads by searching for critical section address: 006f1da8
All other two critical sections are being held by OwningThread and there are no waiting threads for them (no need to worry about them):
BlockedThreads = LockCount - (RecursionCount - 1) = 0 - (1 - 1) = 0
Therefore, the rule of thumb is to look at LockCount values.
Thanks,
Dmitry
July 3rd, 2007 at 10:59 pm
Part 9b:
http://www.dumpanalysis.org/blog/index.php/2007/07/03/crash-dump-analysis-patterns-part-9b/
October 16th, 2007 at 2:45 pm
Another deadlock in IE:
0:000> !locks
CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0
WaiterWoken No
LockCount 3
RecursionCount 2
OwningThread d5a8
EntryCount 0
ContentionCount 5a
*** Locked
CritSec shell32!CMountPoint::_csDL+0 at 7cae42d0
WaiterWoken No
LockCount 1
RecursionCount 1
OwningThread b7b4
EntryCount 0
ContentionCount 7
*** Locked
Scanned 1024 critical sections
0:000> ~*kb 100
. 0 Id: c068.b7b4 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr Args to Child
0013bd0c 7c827d0b 7c83d236 000001d0 00000000 ntdll!KiFastSystemCallRet
0013bd10 7c83d236 000001d0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0013bd4c 7c83d281 000001d0 00000004 00000001 ntdll!RtlpWaitOnCriticalSection+0x1a3
0013bd6c 7c82f20c 7c8877a0 00000000 0013be68 ntdll!RtlEnterCriticalSection+0xa8
0013bda0 7c82f336 00000000 00000000 0013bde8 ntdll!LdrLockLoaderLock+0x133
0013be1c 7c82f2a3 00000001 00000001 00000000 ntdll!LdrGetDllHandleEx+0x94
0013be38 77e65185 00000001 00000000 0013bea0 ntdll!LdrGetDllHandle+0x18
0013be84 77e6528f 0013bea0 00000000 7cae2f60 kernel32!GetModuleHandleForUnicodeString+0x20
0013c2fc 77e65155 00000001 00000002 7c8d8828 kernel32!BasepGetModuleHandleExW+0x17f
0013c314 7c91079e 7c8d8828 7c9107b8 0013c350 kernel32!GetModuleHandleW+0x29
0013c31c 7c9107b8 0013c350 7c91078d 00000001 shell32!IsProcessAnExplorer+0xb
0013c324 7c91078d 00000001 7c91373b 00000018 shell32!IsMainShellProcess2+0x46
0013c32c 7c91373b 00000018 00000000 7cae42d0 shell32!_Shell32LoadedInDesktop+0x7
0013c350 7c913776 00000018 00000000 7cae42d0 shell32!CMountPoint::_IsNetDriveLazyLoadNetDLLs+0x7b
0013c37c 7c9136dc 00000018 00000001 0013c634 shell32!CMountPoint::_GetMountPointDL+0x1c
0013c398 7c96dfd7 00000018 00000001 00000001 shell32!CMountPoint::GetMountPoint+0x46
0013c5e4 7c90f37d 0018e988 00000001 001a0ea8 shell32!CDrivesFolder::GetAttributesOf+0x7b
0013c624 779cc875 0018e9b0 00000001 04002000 shell32!CRegFolder::GetAttributesOf+0x122
0013c648 779cc917 0018e9b0 001e4dc8 04002000 shdocvw!SHGetAttributes+0x53
0013d728 779cd9c8 0013ddac 00193a50 80004005 shdocvw!CNscTree::_OnCDNotify+0x85
0013d754 779cd964 0013ddac 001a06c8 11281f2a shdocvw!CNscTree::_OnNotify+0x2e1
0013d768 779cd8ff 001a06c8 00010090 0000004e shdocvw!CNscTree::OnWinEvent+0x51
0013d798 75eba756 00193a50 00010090 0000004e shdocvw!CNSCBand::OnWinEvent+0x70
0013d7b8 75eba2a2 00193a50 00010090 0000004e browseui!_FwdWinEvent+0x1d
0013d7ec 75eba357 00010090 0000004e 00000064 browseui!CBandSite::_SendToToolband+0x44
0013d818 75ee2a72 0017de98 00010088 00000000 browseui!CBandSite::OnWinEvent+0x143
0013d864 75ee2b32 0017de98 00010088 0000004e browseui!CBrowserBandSite::OnWinEvent+0x14c
0013d890 75ee2a9a 0000004e 00000064 0013ddac browseui!CBaseBar::_CheckForwardWinEvent+0x88
0013d8ac 75ee29dc 0000004e 00000064 0013ddac browseui!CBaseBar::_OnNotify+0x1c
0013d8c8 75ee2965 00010088 0000004e 00000064 browseui!CBaseBar::v_WndProc+0xd4
0013d918 75ee28fa 00010088 0000004e 00000064 browseui!CDockingBar::v_WndProc+0x447
0013d948 75ee2880 00010088 0000004e 00000064 browseui!CBrowserBar::v_WndProc+0x99
0013d96c 7739b6e3 00010088 0000004e 00000064 browseui!CImpWndProc::s_WndProc+0x65
0013d998 7739b874 75ee2841 00010088 0000004e user32!InternalCallWinProc+0x28
0013da10 7739c2d3 00172e34 75ee2841 00010088 user32!UserCallWinProcCheckWow+0x151
0013da4c 7739c337 006172a0 00618f18 00000064 user32!SendMessageWorker+0x4bd
0013da6c 7743b07f 00010088 0000004e 00000064 user32!SendMessageW+0x7f
0013db04 7743b1ef 0013db1c fffffff4 0013ddac comctl32!CCSendNotify+0xc24
0013db40 774a5ab0 00010088 ffffffff fffffff4 comctl32!SendNotifyEx+0x57
0013dbac 774a652d 0001008a 0000004e 00000064 comctl32!CReBar::_WndProc+0x257
0013dbd0 7739b6e3 0001008a 0000004e 00000064 comctl32!CReBar::s_WndProc+0x2c
0013dbfc 7739b874 774a6501 0001008a 0000004e user32!InternalCallWinProc+0x28
0013dc74 7739c2d3 00172e34 774a6501 0001008a user32!UserCallWinProcCheckWow+0x151
0013dcb0 7739c337 00617350 0060a9c0 00000064 user32!SendMessageWorker+0x4bd
0013dcd0 7743b07f 0001008a 0000004e 00000064 user32!SendMessageW+0x7f
0013dd68 7743b10d 001c8900 fffffff4 0013ddac comctl32!CCSendNotify+0xc24
0013dd7c 7748a032 001c8900 00010001 0013ddac comctl32!CICustomDrawNotify+0x2c
0013e070 7748a8bb 001c8900 001d2aa8 01010060 comctl32!TV_DrawItem+0x356
0013e0f4 7748a9ac 00000154 01010060 00000000 comctl32!TV_DrawTree+0x136
0013e158 7745bdd0 001c8900 00000000 0013e21c comctl32!TV_Paint+0x65
0013e1a4 7739b6e3 00010090 0000000f 00000000 comctl32!TV_WndProc+0x6ea
0013e1d0 7739b874 7745b6e6 00010090 0000000f user32!InternalCallWinProc+0x28
0013e248 7739bfce 0015fce4 7745b6e6 00010090 user32!UserCallWinProcCheckWow+0x151
0013e278 7739bf74 7745b6e6 00010090 0000000f user32!CallWindowProcAorW+0x98
0013e298 77431848 7745b6e6 00010090 0000000f user32!CallWindowProcW+0x1b
0013e2b4 77431b9b 00010090 0000000f 00000000 comctl32!CallOriginalWndProc+0x1a
0013e310 77431d5d 001cf0f8 00010090 0000000f comctl32!CallNextSubclassProc+0x3c
0013e334 779cd761 00010090 0000000f 00000000 comctl32!DefSubclassProc+0x46
0013e350 77431b9b 00010090 0000000f 00000000 shdocvw!CNotifySubclassWndProc::_SubclassWndProc+0xa7
0013e3ac 77431d5d 001cf0f8 00010090 0000000f comctl32!CallNextSubclassProc+0x3c
0013e3d0 779cd86f 00010090 0000000f 00000000 comctl32!DefSubclassProc+0x46
0013e41c 779cd7e4 00010090 0000000f 00000000 shdocvw!CNscTree::_SubClassTreeWndProc+0x3ae
0013e43c 77431b9b 00010090 0000000f 00000000 shdocvw!CNscTree::s_SubClassTreeWndProc+0x34
0013e498 77431dc0 001cf0f8 00010090 0000000f comctl32!CallNextSubclassProc+0x3c
0013e4ec 7739b6e3 00010090 0000000f 00000000 comctl32!MasterSubclassProc+0x54
0013e518 7739b874 77431d6c 00010090 0000000f user32!InternalCallWinProc+0x28
0013e590 7739c8b8 0015fce4 77431d6c 00010090 user32!UserCallWinProcCheckWow+0x151
0013e5ec 7739c9c6 00617618 0000000f 00000000 user32!DispatchClientMessage+0xd9
0013e614 7c828536 0013e62c 00000018 0013e750 user32!__fnDWORD+0x24
0013e640 7739cbb2 7739cb75 00010090 0000005e ntdll!KiUserCallbackDispatcher+0x2e
0013e654 77459d14 00010090 00000200 001c8900 user32!NtUserCallHwndLock+0xc
0013e66c 7745bd2d 00000004 016b0055 00000000 comctl32!TV_OnMouseMove+0x62
0013e6bc 7739b6e3 00010090 00000200 00000000 comctl32!TV_WndProc+0x647
0013e6e8 7739b874 7745b6e6 00010090 00000200 user32!InternalCallWinProc+0x28
0013e760 7739bfce 0015fce4 7745b6e6 00010090 user32!UserCallWinProcCheckWow+0x151
0013e790 7739bf74 7745b6e6 00010090 00000200 user32!CallWindowProcAorW+0x98
0013e7b0 77431848 7745b6e6 00010090 00000200 user32!CallWindowProcW+0x1b
0013e7cc 77431b9b 00010090 00000200 00000000 comctl32!CallOriginalWndProc+0x1a
0013e828 77431d5d 001cf0f8 00010090 00000200 comctl32!CallNextSubclassProc+0x3c
0013e84c 779cd761 00010090 00000200 00000000 comctl32!DefSubclassProc+0x46
0013e868 77431b9b 00010090 00000200 00000000 shdocvw!CNotifySubclassWndProc::_SubclassWndProc+0xa7
0013e8c4 77431d5d 001cf0f8 00010090 00000200 comctl32!CallNextSubclassProc+0x3c
0013e8e8 779cd86f 00010090 00000200 00000000 comctl32!DefSubclassProc+0x46
0013e934 779cd7e4 00010090 00000200 00000000 shdocvw!CNscTree::_SubClassTreeWndProc+0x3ae
0013e954 77431b9b 00010090 00000200 00000000 shdocvw!CNscTree::s_SubClassTreeWndProc+0x34
0013e9b0 77431dc0 001cf0f8 00010090 00000200 comctl32!CallNextSubclassProc+0x3c
0013ea04 7739b6e3 00010090 00000200 00000000 comctl32!MasterSubclassProc+0x54
0013ea30 7739b874 77431d6c 00010090 00000200 user32!InternalCallWinProc+0x28
0013eaa8 7739ba92 0015fce4 77431d6c 00010090 user32!UserCallWinProcCheckWow+0x151
0013eb10 7739bad0 0013eb50 00000000 0013eb38 user32!DispatchMessageWorker+0x327
0013eb20 75ed1410 0013eb50 00000000 00176388 user32!DispatchMessageW+0xf
0013eb38 75ed14fc 0013eb50 0013ee50 00000000 browseui!TimedDispatchMessage+0x33
0013ed98 75ec1c83 0015f7e8 0013ee50 0015f7e8 browseui!BrowserThreadProc+0x336
0013ee24 75ec61ef 0015f7e8 0015f7e8 00000000 browseui!BrowserProtectedThreadProc+0x44
0013fea8 779ba3a6 0015f7e8 00000001 00000000 browseui!SHOpenFolderWindow+0x22c
0013fec8 0040243d 00152552 00020d02 ffffffff shdocvw!IEWinMain+0x129
0013ff1c 00402744 00400000 00000000 00152552 iexplore!WinMain+0x316
0013ffc0 77e6f23b 00000000 00000000 7ffde000 iexplore!WinMainCRTStartup+0x182
0013fff0 00000000 004025c2 00000000 78746341 kernel32!BaseProcessStart+0x23
1 Id: c068.d71c Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr Args to Child
00d4fea0 7c827cfb 7c80e5bb 00000002 00d4fef0 ntdll!KiFastSystemCallRet
00d4fea4 7c80e5bb 00000002 00d4fef0 00000001 ntdll!NtWaitForMultipleObjects+0xc
00d4ff48 7c80e4a2 00000002 00d4ff70 00000000 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
00d4ffb8 77e64829 00000000 00000000 00000000 ntdll!EtwpEventPump+0x27f
00d4ffec 00000000 7c80e1fa 00000000 00000000 kernel32!BaseThreadStart+0x34
2 Id: c068.cba4 Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr Args to Child
012bfe18 7c82783b 77c885ac 000001c4 012bff74 ntdll!KiFastSystemCallRet
012bfe1c 77c885ac 000001c4 012bff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc
012bff84 77c88792 012bffac 77c8872d 00153cf0 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
012bff8c 77c8872d 00153cf0 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
012bffac 77c7b110 00167030 012bffec 77e64829 rpcrt4!BaseCachedThreadRoutine+0x9d
012bffb8 77e64829 00172088 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
012bffec 00000000 77c7b0f5 00172088 00000000 kernel32!BaseThreadStart+0x34
3 Id: c068.8604 Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr Args to Child
013bfe28 7c827d0b 7c83d236 000001d0 00000000 ntdll!KiFastSystemCallRet
013bfe2c 7c83d236 000001d0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
013bfe68 7c83d281 000001d0 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3
013bfe88 7c839844 7c8877a0 00000000 77670000 ntdll!RtlEnterCriticalSection+0xa8
013bff90 77e52860 77670000 77670000 00171698 ntdll!LdrUnloadDll+0x35
013bffa4 776b171d 77670000 00000000 00000000 kernel32!FreeLibraryAndExitThread+0x38
013bffb8 77e64829 00171698 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x39
013bffec 00000000 776b16e4 00171698 00000000 kernel32!BaseThreadStart+0x34
4 Id: c068.d6dc Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr Args to Child
016dfd24 7c827cfb 77e6202c 00000005 016dfd74 ntdll!KiFastSystemCallRet
016dfd28 77e6202c 00000005 016dfd74 00000001 ntdll!NtWaitForMultipleObjects+0xc
016dfdd0 7739bbd1 00000005 016dfdf8 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
016dfe2c 7c919b2e 00000004 016dfe54 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
016dff50 7c8f7ada 77da3f12 00000000 00000000 shell32!CChangeNotify::_MessagePump+0x3b
016dff54 77da3f12 00000000 00000000 00000000 shell32!CChangeNotify::ThreadProc+0x1e
016dffb8 77e64829 00000000 00000000 00000000 shlwapi!WrapperThreadProc+0x94
016dffec 00000000 77da3ea5 0013dea8 00000000 kernel32!BaseThreadStart+0x34
5 Id: c068.caf4 Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr Args to Child
01b1fdb4 7c827cfb 77e6202c 00000002 01b1fe04 ntdll!KiFastSystemCallRet
01b1fdb8 77e6202c 00000002 01b1fe04 00000001 ntdll!NtWaitForMultipleObjects+0xc
01b1fe60 7739bbd1 00000002 01b1fe88 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
01b1febc 6c296601 00000001 01b1fef0 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
01b1fedc 6c29684b 000004ff ffffffff 00000001 duser!CoreSC::Wait+0x3a
01b1ff10 6c28f9e6 01b1ff50 00000000 00000000 duser!CoreSC::xwProcessNL+0xab
01b1ff30 6c28bce1 01b1ff50 00000000 00000000 duser!GetMessageExA+0x44
01b1ff84 77bcb530 00000000 00000000 00000000 duser!ResourceManager::SharedThreadProc+0xb6
01b1ffb8 77e64829 000385f0 00000000 00000000 msvcrt!_endthreadex+0xa3
01b1ffec 00000000 77bcb4bc 000385f0 00000000 kernel32!BaseThreadStart+0x34
6 Id: c068.d624 Suspend: 1 Teb: 7ffd7000 Unfrozen
ChildEBP RetAddr Args to Child
01c9ff9c 7c826f4b 7c83d424 00000001 01c9ffb0 ntdll!KiFastSystemCallRet
01c9ffa0 7c83d424 00000001 01c9ffb0 00000000 ntdll!NtDelayExecution+0xc
01c9ffb8 77e64829 00000000 00000000 00000000 ntdll!RtlpTimerThread+0x47
01c9ffec 00000000 7c83d3dd 00000000 00000000 kernel32!BaseThreadStart+0x34
7 Id: c068.b4e0 Suspend: 1 Teb: 7ffd6000 Unfrozen
ChildEBP RetAddr Args to Child
01d9fd58 7c827d0b 7c83d236 000001d0 00000000 ntdll!KiFastSystemCallRet
01d9fd5c 7c83d236 000001d0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
01d9fd98 7c83d281 000001d0 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3
01d9fdb8 7c839844 7c8877a0 75eb8b7c 75eb0000 ntdll!RtlEnterCriticalSection+0xa8
01d9fec0 77e6b1bb 75eb0000 75eb0000 001e2f98 ntdll!LdrUnloadDll+0x35
01d9fed4 77da4c1c 75eb0000 0020eec8 77da591b kernel32!FreeLibrary+0x41
01d9feec 7c83a827 0020eec8 7c889080 001e4ec0 shlwapi!ExecuteWorkItem+0x28
01d9ff44 7c83aa0b 77da591b 0020eec8 00000000 ntdll!RtlpWorkerCallout+0x71
01d9ff64 7c83aa82 00000000 0020eec8 001e4ec0 ntdll!RtlpExecuteWorkerRequest+0x4f
01d9ff78 7c839f60 7c83a9ca 00000000 0020eec8 ntdll!RtlpApcCallout+0x11
01d9ffb8 77e64829 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x61
01d9ffec 00000000 7c839efb 00000000 00000000 kernel32!BaseThreadStart+0x34
8 Id: c068.d5a8 Suspend: 1 Teb: 7ffd5000 Unfrozen
ChildEBP RetAddr Args to Child
01fbb41c 7c827d0b 7c83d236 00000468 00000000 ntdll!KiFastSystemCallRet
01fbb420 7c83d236 00000468 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
01fbb45c 7c83d281 00000468 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3
01fbb47c 7c9136c9 7cae42d0 001c97b0 80070003 ntdll!RtlEnterCriticalSection+0xa8
01fbb494 7c913b75 0000000c 00000000 00000001 shell32!CMountPoint::GetMountPoint+0x33
01fbb4c8 7c91358d 01fbb4fc 0000000c 00000000 shell32!CDrivesFolder::_FillIDDrive+0x5c
01fbb52c 7c9109e7 0018e988 00000000 001c97b0 shell32!CDrivesFolder::ParseDisplayName+0x9f
01fbb594 7c9119ff 0018e9b0 00000000 001c97b0 shell32!CRegFolder::ParseDisplayName+0x93
01fbb5bc 7c910bb8 00000000 001a8e30 00000000 shell32!CDesktopFolder::_ChildParseDisplayName+0x22
01fbb60c 7c9109e7 0017cde0 00000000 001c97b0 shell32!CDesktopFolder::ParseDisplayName+0x7e
01fbb674 7c910a9b 0015f058 00000000 001c97b0 shell32!CRegFolder::ParseDisplayName+0x93
01fbb6ac 7c911ab4 00000000 00000000 00000000 shell32!SHParseDisplayName+0xa3
01fbb6d0 7c911a6e 01fbbe60 00000000 00000002 shell32!ILCreateFromPathEx+0x3d
01fbb6ec 7c911a4b 01fbbe60 01fbb700 00000000 shell32!SHILCreateFromPath+0x17
01fbb704 7c95e055 01fbbe60 00000104 01fbc0a0 shell32!ILCreateFromPathW+0x18
01fbbb84 7c9ef49d 01fbbe60 00000000 01fbbbac shell32!SHGetFileInfoW+0x117
01fbc06c 01b4d195 01fbc200 00000000 01fbc0a0 shell32!SHGetFileInfoA+0x6a
WARNING: Stack unwind information not available. Following frames may be wrong.
01fbc0a4 01b54a20 0000073c 02541f28 00000000 issftran!SSCopyFile+0x27ad
00000000 00000000 00000000 00000000 00000000 issftran!DllUnregisterServer+0x70ad
9 Id: c068.d750 Suspend: 1 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr Args to Child
0228ff7c 7c8277db 71b25914 000004b4 0228ffc0 ntdll!KiFastSystemCallRet
0228ff80 71b25914 000004b4 0228ffc0 0228ffb4 ntdll!ZwRemoveIoCompletion+0xc
0228ffb8 77e64829 71b259de 00000000 00000000 mswsock!SockAsyncThread+0x69
0228ffec 00000000 71b258ab 001fcd20 00000000 kernel32!BaseThreadStart+0x34
0:000> du 7c8d8828
7c8d8828 "EXPLORER.EXE"
0:000> da 01fbc200
01fbc200 "M:\WINDOWS"
October 31st, 2007 at 1:05 pm
There is also !cs WinDbg extension where !cs -l lists all locked sections with stack traces and !cs -t shows critical section tree. For the latter you need to enable Application Verifier using gflags.exe or set 0×100 in registry for your image:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options
GlobalFlag=0×00000100
December 20th, 2007 at 1:36 pm
Hello, Dmitry!
Thank you for your blog! It helps me very much
One question for this article… Is there any way to walk through the situation if
thread-owner of critical section is already dead and absent in call-stack?
0:022> !locks
CritSec ole32!g_mxsSingleThreadOle+18 at 76a40664
WaiterWoken No
LockCount 4
RecursionCount 1
OwningThread 1ec
EntryCount 0
ContentionCount 6
*** Locked
P.S. as you can see, this critical section is not mine
December 20th, 2007 at 4:33 pm
Thanks! I tried in the past and what I found is that virtual memory for TEB was decommitted already and therefore no data for that stack start address and its limit was available. I might guess that stack region itself is decommitted during thread termination so we cannot use memory search here to find its raw stack data. Perhaps some test application might clarify here.
Live debugging might help here with scripts to set conditional breakpoints and saving dumps automatically upon some condition. This is interesting and I will explore it in some detail in my future posts.
March 12th, 2008 at 4:44 pm
[…] Deadlock (critical sections) […]
June 3rd, 2008 at 1:58 am
Hi,
I’m trying to figure out the problem described above - the thread owning the (user-mode loader lock) critical section isn’t around any more, so the threads waiting on it aren’t going to wake up any time soon. You said you’d explore it further - did you get a chance to do so yet?
Thanks.
July 22nd, 2008 at 4:14 pm
[…] When a process experienced an unhandled exception what were the possible reasons for a postmortem debugger not saving a crash dump? One of them will be illustrated in this post. The process AppA was hanging and causing another process AppB to hang too (see Coupled Processes pattern). If we look at AppA locked critical sections we would see a loader deadlock: […]
September 18th, 2008 at 6:26 am
[…] 원문 링크: http://www.dumpanalysis.org/blog/index.php/2007/02/09/crash-dump-analysis-patterns-part-9a/ […]
October 25th, 2008 at 7:04 am
[…] involving mixed objects in kernel space. Previously we discussed deadlock patterns involving critical sections in user space, executive resources in kernel space, mixed objects in user space and […]
February 17th, 2009 at 9:29 am
[…] Deadlock (critical sections) […]
March 26th, 2010 at 12:08 pm
[…] Today we introduce an icon for Deadlock (critical sections) pattern: […]
June 8th, 2011 at 9:29 am
Could you share me more about the deadlock caused by critical section and kernel object?
How to debug into this scenario?
For example,
ThreadA:
EnterCriticalSection1
WaitForSingleObject2 (Event)
ThreadB:
EnterCriticalSection1
Event Reset/Activate
now, if threada get run firstly, enter critical section1, but need to wait the event, threadb want to reset the event, but need to require access to critical section.
how to detect such deadlock in the dump file?
April 24th, 2013 at 3:42 pm
Hi Dmitry,
I’m facing a problem with hang threads. when I executed the command !locks I got this:
0:000> !locks
CritSec +83eb6d10 at 83eb6d10
WaiterWoken No
LockCount -452697857
RecursionCount 36294152
OwningThread a
EntryCount 1379620
ContentionCount 1379620
*** Locked
CritSec +840d6d10 at 840d6d10
WaiterWoken Yes
LockCount -60090439
RecursionCount 1093776890
OwningThread ffffffffadcb569d
EntryCount 0
ContentionCount 0
*** Locked
CritSec +840d6d10 at 840d6d10
WaiterWoken Yes
LockCount -60090439
RecursionCount 1093776890
OwningThread ffffffffadcb569d
EntryCount 0
ContentionCount 0
*** Locked
CritSec +86156d10 at 86156d10
WaiterWoken Yes
LockCount -278997059
RecursionCount 1224415142
OwningThread ffffffff9e8e4272
EntryCount 0
ContentionCount 0
*** Locked
CritSec +83c96d10 at 83c96d10
WaiterWoken Yes
LockCount 520027693
RecursionCount 1097194433
OwningThread ffffffff8c000036
EntryCount 0
ContentionCount 0
*** Locked
CritSec +83c96d10 at 83c96d10
WaiterWoken Yes
LockCount 520027693
RecursionCount 1097194433
OwningThread ffffffff8c000036
EntryCount 0
ContentionCount 0
*** Locked
Scanned 1223 critical sections
I don’t know what to do with this result, As you can see here, LockCount and RecursionCount have strange numbers.
Thanks.
April 25th, 2013 at 7:01 pm
You might have corrupt critical section list.
http://www.dumpanalysis.org/blog/index.php/2008/07/11/crash-dump-analysis-patterns-part-71/
Try also !cs -l -o -s command. Sometimes if you dump all sections you can see where corruption starts like !locks -v or !cs
October 5th, 2014 at 3:49 pm
0:000:x86> ~*kv
. 0 Id: 2e24.35f8 Suspend: 0 Teb: 7efdb000 Unfrozen
ChildEBP RetAddr Args to Child
0015cedc 77748e44 000004b8 00000000 00000000 ntdll_77710000!ZwWaitForSingleObject+0×15 (FPO: [3,0,0])
0015cf40 77748d28 00000000 00000000 000035f8 ntdll_77710000!RtlpWaitOnCriticalSection+0×13e (FPO: [Non-Fpo])
0015cf68 558f829b 00e531d8 d3c25775 058f0a0c ntdll_77710000!RtlEnterCriticalSection+0×150 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0015d4a4 558f9e66 d3c256e1 000000df 00000000 libdjvulibre!DJVU::DjVuDocument::process_threqs+0×6b
0015d530 558cdea5 0015d57c 000000df 00000001 libdjvulibre!DJVU::DjVuDocument::get_thumbnail+0×8c6
0015d594 0103d8d1 00c6a1e0 000000df 00000000 libdjvulibre!ddjvu_thumbnail_status+0×115
0015d604 55107cd9 00c3c5a8 00000000 00000006 djview+0×4d8d1
0015d658 53f3fc29 00000000 00c3c5a8 00b8ed18 QtCore4!QMetaCallEvent::placeMetaCall+0×19
0015d8a0 550f948d 00c3c5a8 00eeb598 d3c25ad6 QtGui4!QApplicationPrivate::notify_helper+0xb9
0015d8e0 550fb07f 00b8e640 00eeb598 d3c25b12 QtCore4!QCoreApplication::notifyInternal+0×8d
0015d924 5511e835 00000000 00000000 00b8e640 QtCore4!QCoreApplicationPrivate::sendPostedEvents+0×1cf
00000000 00000000 00000000 00000000 00000000 QtCore4!QEventDispatcherWin32::event+0×555
1 Id: 2e24.1390 Suspend: 0 Teb: 7efd8000 Unfrozen
ChildEBP RetAddr Args to Child
0296f884 7344a41c 00000001 0296f8e4 00000001 ntdll_77710000!NtWaitForMultipleObjects+0×15 (FPO: [5,0,0])
0296f92c 7702338a 00000000 0296f978 77749f72 winmm!timeThread+0×3c (FPO: [Non-Fpo])
0296f938 77749f72 00000000 55277637 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0296f978 77749f45 7344a3e0 00000000 00000000 ntdll_77710000!__RtlUserThreadStart+0×70 (FPO: [Non-Fpo])
0296f990 00000000 7344a3e0 00000000 00000000 ntdll_77710000!_RtlUserThreadStart+0×1b (FPO: [Non-Fpo])
2 Id: 2e24.37a0 Suspend: 0 Teb: 7efd5000 Unfrozen
ChildEBP RetAddr Args to Child
02c3facc 77762f91 00000003 005fb610 00000001 ntdll_77710000!NtWaitForMultipleObjects+0×15 (FPO: [5,0,0])
02c3fc60 7702338a 00000000 02c3fcac 77749f72 ntdll_77710000!TppWaiterpThread+0×33d (FPO: [Non-Fpo])
02c3fc6c 77749f72 005fb5e0 557273e3 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
02c3fcac 77749f45 77762e65 005fb5e0 00000000 ntdll_77710000!__RtlUserThreadStart+0×70 (FPO: [Non-Fpo])
02c3fcc4 00000000 77762e65 005fb5e0 00000000 ntdll_77710000!_RtlUserThreadStart+0×1b (FPO: [Non-Fpo])
3 Id: 2e24.2150 Suspend: 0 Teb: 7ef4a000 Unfrozen
ChildEBP RetAddr Args to Child
0300f940 77763392 0000039c 0300f9f4 54b175ef ntdll_77710000!NtWaitForWorkViaWorkerFactory+0×12 (FPO: [2,0,0])
0300faa0 7702338a 005fa788 0300faec 77749f72 ntdll_77710000!TppWorkerThread+0×216 (FPO: [Non-Fpo])
0300faac 77749f72 005fa788 54b175a3 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0300faec 77749f45 77763e85 005fa788 00000000 ntdll_77710000!__RtlUserThreadStart+0×70 (FPO: [Non-Fpo])
0300fb04 00000000 77763e85 005fa788 00000000 ntdll_77710000!_RtlUserThreadStart+0×1b (FPO: [Non-Fpo])
4 Id: 2e24.3478 Suspend: 0 Teb: 7ef4d000 Unfrozen
ChildEBP RetAddr Args to Child
0320f3bc 77748e44 000004c4 00000000 00000000 ntdll_77710000!ZwWaitForSingleObject+0×15 (FPO: [3,0,0])
0320f420 77748d28 00000000 00000000 058f0a18 ntdll_77710000!RtlpWaitOnCriticalSection+0×13e (FPO: [Non-Fpo])
0320f448 558f8750 04f3538c d0f77a55 058dbb18 ntdll_77710000!RtlEnterCriticalSection+0×150 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0320f984 558fa26c d0f77a41 058dbb18 04ffb618 libdjvulibre!DJVU::DjVuDocument::process_threqs+0×520
0320f9b0 5591ec70 058dbb18 00000042 00000001 libdjvulibre!DJVU::DjVuDocument::notify_file_flags_changed+0xdc
0320fa00 55900ac1 058dbb18 00000042 00000001 libdjvulibre!DJVU::DjVuPortcaster::notify_file_flags_changed+0×80
0320faf4 559005a7 d0f778f1 00000000 00000000 libdjvulibre!DJVU::DjVuFile::decode_func+0×4c1
0320fb20 55945bf2 058dbb18 d0f778b1 00000000 libdjvulibre!DJVU::DjVuFile::static_decode_func+0×87
0320fb60 55b4c556 04d95f90 d0f779b8 00000000 libdjvulibre!DJVU::GNativeString::setat+0×282
0320fb98 55b4c600 00000000 0320fbb0 7702338a msvcr100!_endthreadex+0×3f (FPO: [Non-Fpo])
0320fba4 7702338a 04ad13d8 0320fbf0 77749f72 msvcr100!_endthreadex+0xce (FPO: [Non-Fpo])
0320fbb0 77749f72 04ad13d8 549174bf 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0320fbf0 77749f45 55b4c59c 04ad13d8 00000000 ntdll_77710000!__RtlUserThreadStart+0×70 (FPO: [Non-Fpo])
0320fc08 00000000 55b4c59c 04ad13d8 00000000 ntdll_77710000!_RtlUserThreadStart+0×1b (FPO: [Non-Fpo])
0:000:x86> !cs -l -o -s
—————————————–
DebugInfo = 0×00000000005f05c0
Critical section = 0×0000000000e531d8 (+0xE531D8)
LOCKED
LockCount = 0×1
WaiterWoken = No
OwningThread = 0×0000000000003478
RecursionCount = 0×1
LockSemaphore = 0×4B8
SpinCount = 0×0000000000000000
OwningThread DbgId = ~4s
OwningThread Stack =
ChildEBP RetAddr Args to Child
0320f3bc 77748e44 000004c4 00000000 00000000 ntdll_77710000!ZwWaitForSingleObject+0×15 (FPO: [3,0,0])
0320f420 77748d28 00000000 00000000 058f0a18 ntdll_77710000!RtlpWaitOnCriticalSection+0×13e (FPO: [Non-Fpo])
0320f448 558f8750 04f3538c d0f77a55 058dbb18 ntdll_77710000!RtlEnterCriticalSection+0×150 (FPO: [Non-Fpo])
0320f984 558fa26c d0f77a41 058dbb18 04ffb618 libdjvulibre!DJVU::DjVuDocument::process_threqs+0×520
0320f9b0 5591ec70 058dbb18 00000042 00000001 libdjvulibre!DJVU::DjVuDocument::notify_file_flags_changed+0xdc
0320fa00 55900ac1 058dbb18 00000042 00000001 libdjvulibre!DJVU::DjVuPortcaster::notify_file_flags_changed+0×80
0320faf4 559005a7 d0f778f1 00000000 00000000 libdjvulibre!DJVU::DjVuFile::decode_func+0×4c1
0320fb20 55945bf2 058dbb18 d0f778b1 00000000 libdjvulibre!DJVU::DjVuFile::static_decode_func+0×87
0320fb60 55b4c556 04d95f90 d0f779b8 00000000 libdjvulibre!DJVU::GNativeString::setat+0×282
0320fb98 55b4c600 00000000 0320fbb0 7702338a msvcr100!_endthreadex+0×3f (FPO: [Non-Fpo])
0320fba4 7702338a 04ad13d8 0320fbf0 77749f72 msvcr100!_endthreadex+0xce (FPO: [Non-Fpo])
0320fbb0 77749f72 04ad13d8 549174bf 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0320fbf0 77749f45 55b4c59c 04ad13d8 00000000 ntdll_77710000!__RtlUserThreadStart+0×70 (FPO: [Non-Fpo])
0320fc08 00000000 55b4c59c 04ad13d8 00000000 ntdll_77710000!_RtlUserThreadStart+0×1b (FPO: [Non-Fpo])
${$ntdllwsym}!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
—————————————–
DebugInfo = 0×0000000000625d80
Critical section = 0×0000000004f3538c (+0×4F3538C)
LOCKED
LockCount = 0×1
WaiterWoken = No
OwningThread = 0×00000000000035f8
RecursionCount = 0×1
LockSemaphore = 0×4C4
SpinCount = 0×0000000000000000
OwningThread DbgId = ~0s
OwningThread Stack =
ChildEBP RetAddr Args to Child
0015cedc 77748e44 000004b8 00000000 00000000 ntdll_77710000!ZwWaitForSingleObject+0×15 (FPO: [3,0,0])
0015cf40 77748d28 00000000 00000000 000035f8 ntdll_77710000!RtlpWaitOnCriticalSection+0×13e (FPO: [Non-Fpo])
0015cf68 558f829b 00e531d8 d3c25775 058f0a0c ntdll_77710000!RtlEnterCriticalSection+0×150 (FPO: [Non-Fpo])
0015d4a4 558f9e66 d3c256e1 000000df 00000000 libdjvulibre!DJVU::DjVuDocument::process_threqs+0×6b
0015d530 558cdea5 0015d57c 000000df 00000001 libdjvulibre!DJVU::DjVuDocument::get_thumbnail+0×8c6
0015d594 0103d8d1 00c6a1e0 000000df 00000000 libdjvulibre!ddjvu_thumbnail_status+0×115
0015d604 55107cd9 00c3c5a8 00000000 00000006 djview+0×4d8d1
0015d658 53f3fc29 00000000 00c3c5a8 00b8ed18 QtCore4!QMetaCallEvent::placeMetaCall+0×19
0015d8a0 550f948d 00c3c5a8 00eeb598 d3c25ad6 QtGui4!QApplicationPrivate::notify_helper+0xb9
0015d8e0 550fb07f 00b8e640 00eeb598 d3c25b12 QtCore4!QCoreApplication::notifyInternal+0×8d
0015d924 5511e835 00000000 00000000 00b8e640 QtCore4!QCoreApplicationPrivate::sendPostedEvents+0×1cf
00000000 00000000 00000000 00000000 00000000 QtCore4!QEventDispatcherWin32::event+0×555
${$ntdllwsym}!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
September 30th, 2015 at 4:29 pm
Often, we can preliminary suppose a critical section deadlock if all locked critical section owner threads are waiting for critical sections (as seen from !cs -l -o -s WinDbg command output and corresponding stack traces, k or !thread).