Crash Dump Analysis Patterns (Part 42d)
Another example of Wait Chain pattern for objects with ownership semantics is seen in kernel and complete memory dumps where threads are waiting for thread objects. A thread object is a synchronization object whose owner is a thread so they can be easily identified. For example, the main application thread is waiting for an object:
1: kd> !thread 8818e660 16
THREAD 8818e660 Cid 1890.1c50 Teb: 7ffdf000 Win32Thread: b8411008 WAIT: (Unknown) UserMode Non-Alertable
87d569d8 Thread
8818e6d8 NotificationTimer
Not impersonating
DeviceMap e10008d8
Owning Process 87db5d88 Image: App.exe
Wait Start TickCount 299006 Ticks: 255 (0:00:00:03.984)
Context Switch Count 1208 LargeStack
UserTime 00:00:00.203
KernelTime 00:00:00.203
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b42c3000 Current b42c2c60 Base b42c3000 Limit b42be000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0
ChildEBP RetAddr Args to Child
b42c2c78 80833e8d 8818e660 8818e708 00000003 nt!KiSwapContext+0×26
b42c2ca4 80829b74 00000000 b42c2d14 00000000 nt!KiSwapThread+0×2e5
b42c2cec 8093b034 87d569d8 00000006 00804c01 nt!KeWaitForSingleObject+0×346
b42c2d50 8088ac4c 000001ec 00000000 b42c2d14 nt!NtWaitForSingleObject+0×9a
b42c2d50 7c8285ec 000001ec 00000000 b42c2d14 nt!KiFastCallEntry+0xfc
0006fde4 7c827d0b 77e61d1e 000001ec 00000000 ntdll!KiFastSystemCallRet
0006fde8 77e61d1e 000001ec 00000000 0006fe2c ntdll!NtWaitForSingleObject+0xc
0006fe58 77e61c8d 000001ec 00001388 00000000 kernel32!WaitForSingleObjectEx+0xac
0006fe6c 01039308 000001ec 00001388 00000000 kernel32!WaitForSingleObject+0×12
0006fe94 010204ac 0007cc00 00000001 00000002 App!WaitForNotifyList+0xf2
[…]
However that object is a thread too:
THREAD 87d569d8 Cid 1890.1ec0 Teb: 7ffd9000 Win32Thread: b869ba48 WAIT: (Unknown) UserMode Non-Alertable
8a1f8870 Thread
Therefore, we see that thread 8818e660 is waiting for another thread 87d569d8 which belongs to the same process with PID 1890 and thread 87d569d8 itself is waiting for thread 8a1f8870 which has the following stack trace:
1: kd> !thread 8a1f8870 16
THREAD 8a1f8870 Cid 1890.07d8 Teb: 7ff95000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
8a0ce4c0 NotificationEvent
886f1870 NotificationEvent
Not impersonating
DeviceMap e10008d8
Owning Process 87db5d88 Image: App.exe
Wait Start TickCount 292599 Ticks: 6662 (0:00:01:44.093)
Context Switch Count 17
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address Dll!StartMonitoring (0×758217b8)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b6d4f000 Current b6d4e900 Base b6d4f000 Limit b6d4c000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr Args to Child
b6d4e918 80833e8d 8a1f8870 00000002 00140000 nt!KiSwapContext+0×26
b6d4e944 808295ab 8a1f8870 00000002 00000000 nt!KiSwapThread+0×2e5
b6d4e978 8093b290 00000002 b6d4eaac 00000001 nt!KeWaitForMultipleObjects+0×3d7
b6d4ebf4 8093b3f2 00000002 b6d4ec1c 00000001 nt!ObpWaitForMultipleObjects+0×202
b6d4ed48 8088ac4c 00000002 026bfc08 00000001 nt!NtWaitForMultipleObjects+0xc8
b6d4ed48 7c8285ec 00000002 026bfc08 00000001 nt!KiFastCallEntry+0xfc
026bfbb8 7c827cfb 77e6202c 00000002 026bfc08 ntdll!KiFastSystemCallRet
026bfbbc 77e6202c 00000002 026bfc08 00000001 ntdll!NtWaitForMultipleObjects+0xc
026bfc64 77e62fbe 00000002 026bfca4 00000000 kernel32!WaitForMultipleObjectsEx+0×11a
026bfc80 6554a01f 00000002 026bfca4 00000000 kernel32!WaitForMultipleObjects+0×18
026bfcfc 758237a3 cd050002 ffffffff 026bfd4c Dll!GetStatusChange+0×7bf
026bffb8 77e64829 75833120 00000000 00000000 Dll!StartMonitoring+0×14b
026bffec 00000000 758217b8 75833120 00000000 kernel32!BaseThreadStart+0×34
Thread 8a1f8870 is waiting for two notification events disjointly which is confirmed by dumping WaitForMultipleObjects arguments. Neither of them is in signaled state and one is a named event “MyEventObject”:
1: kd> dd 026bfc08 l2
026bfc08 0000008c 00000084
1: kd> !handle 0000008c
processor number 1, process 87db5d88
PROCESS 87db5d88 SessionId: 4 Cid: 1890 Peb: 7ffdc000 ParentCid: 01d0
DirBase: cfe438e0 ObjectTable: e178c228 HandleCount: 439.
Image: App.exe
Handle table at e50d2000 with 439 Entries in use
008c: Object: 8a0ce4c0 GrantedAccess: 001f0003 Entry: e50d2118
Object: 8a0ce4c0 Type: (8b26ec00) Event
ObjectHeader: 8a0ce4a8 (old version)
HandleCount: 1 PointerCount: 3
1: kd> !handle 00000084
processor number 1, process 87db5d88
PROCESS 87db5d88 SessionId: 4 Cid: 1890 Peb: 7ffdc000 ParentCid: 01d0
DirBase: cfe438e0 ObjectTable: e178c228 HandleCount: 439.
Image: App.exe
Handle table at e50d2000 with 439 Entries in use
0084: Object: 886f1870 GrantedAccess: 001f0003 (Inherit) Entry: e50d2108
Object: 886f1870 Type: (8b26ec00) Event
ObjectHeader: 886f1858 (old version)
HandleCount: 1 PointerCount: 4
Directory Object: e43ee320 Name: MyEventObject
1: kd> dt _DISPATCHER_HEADER 8a0ce4c0
cutildll!_DISPATCHER_HEADER
+0×000 Type : 0 ”
+0×001 Absolute : 0 ”
+0×002 Size : 0×4 ”
+0×003 Inserted : 0 ”
+0×003 DebugActive : 0 ”
+0×000 Lock : 262144
+0×004 SignalState : 0
+0×008 WaitListHead : _LIST_ENTRY [ 0×88519d18 - 0×8a1f8918 ]
1: kd> dt _DISPATCHER_HEADER 886f1870
cutildll!_DISPATCHER_HEADER
+0×000 Type : 0 ”
+0×001 Absolute : 0 ”
+0×002 Size : 0×4 ”
+0×003 Inserted : 0 ”
+0×003 DebugActive : 0 ”
+0×000 Lock : 262144
+0×004 SignalState : 0
+0×008 WaitListHead : _LIST_ENTRY [ 0×88519d30 - 0×8a1f8930 ]
Here is a diagram showing this wait chain:
- Dmitry Vostokov @ DumpAnalysis.org -
February 16th, 2010 at 12:39 am
[…] of a wait chain involving process objects. This Wait Chain pattern variation is similar to threads waiting for thread objects. When looking at stack trace collection from a complete memory dump file we see several threads in […]
August 13th, 2010 at 6:48 pm
[…] that is waiting for a mutant owned by thread 866d63e8 (this is equivalent that thread 85b209d0 is waiting for thread […]
September 20th, 2010 at 3:02 pm
[…] Experts Magazine Online Today we introduce an icon for Wait Chain (thread objects) […]
September 22nd, 2010 at 10:31 am
[…] Wait Chain (thread objects) […]