Software Diagnostics Library

Now we illustrate a synchronization block deadlock pattern in managed code. Here we can use either manual !syncblk WinDbg command coupled stack trace and disassembly analysis or SOSEX extension !dlk command (which automates the whole detection process).

0:011> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
373 052cbf1c            3         1 08f69280   bc0  14   0a1ffd84 System.String
375 052cbd3c            3         1 08f68728   b6c  12   0a1ffd4c System.String

0:011> ~12s
[…]

0:012> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
09c8ebd0 79ed98fd ntdll!KiFastSystemCallRet
09c8ec38 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
09c8ec58 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait+0x3c
09c8ecdc 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
09c8ed2c 79ed9a62 mscorwks!Thread::DoAppropriateWait+0x40
09c8ed88 79e78944 mscorwks!CLREvent::WaitEx+0xf7
09c8ed9c 79ed7b37 mscorwks!CLREvent::Wait+0x17
09c8ee28 79ed7a9e mscorwks!AwareLock::EnterEpilog+0x8c
09c8ee44 79ebd7e4 mscorwks!AwareLock::Enter+0x61
09c8eee4 074c1f38 mscorwks!JIT_MonEnterWorker_Portable+0xb3
09c8ef0c 793b0d1f 0×74c1f38
09c8ef14 79373ecd mscorlib_ni+0×2f0d1f
09c8ef28 793b0c68 mscorlib_ni+0×2b3ecd
09c8ef40 79e7c74b mscorlib_ni+0×2f0c68
09c8ef50 79e7c6cc mscorwks!CallDescrWorker+0×33
09c8efd0 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
09c8f110 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
09c8f12c 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
09c8f140 79fc58cd mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
09c8f328 79ef3207 mscorwks!ThreadNative::KickOffThread_Worker+0×190
09c8f33c 79ef31a3 mscorwks!Thread::DoADCallBack+0×32a
09c8f3d0 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
09c8f40c 79f01723 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
09c8f41c 79f02a5d mscorwks!Thread::RaiseCrossContextException+0×434
09c8f4cc 79f02ab7 mscorwks!Thread::DoADCallBack+0xda
09c8f4e8 79ef31a3 mscorwks!Thread::DoADCallBack+0×310
09c8f57c 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
09c8f5b8 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
09c8f5e0 79fc57b1 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
09c8f5f8 79fc56ac mscorwks!ManagedThreadBase::KickOff+0×13
09c8f694 79f95a2e mscorwks!ThreadNative::KickOffThread+0×269
09c8fd34 76573833 mscorwks!Thread::intermediateThreadProc+0×49
09c8fd40 77c1a9bd kernel32!BaseThreadInitThunk+0xe
09c8fd80 00000000 ntdll!LdrInitializeThunk+0×4d

0:012> ub 074c1f38
074c1f11 eb10            jmp     074c1f23
074c1f13 8b0df8927b02    mov     ecx,dword ptr ds:[27B92F8h]
074c1f19 e8367ef271      call    mscorlib_ni+0×329d54 (793e9d54)
074c1f1e e89272a472      call    mscorwks!JIT_EndCatch (79f091b5)
074c1f23 b9d0070000      mov     ecx,7D0h
074c1f28 e8c432b072      call    mscorwks!ThreadNative::Sleep (79fc51f1)
074c1f2d 8b0d88dc7b02    mov     ecx,dword ptr ds:[27BDC88h]
074c1f33 e811389b72      call    mscorwks!JIT_MonEnterWorker (79e75749)

0:012> dp 27BDC88h l1
027bdc88  0a1ffd84

0:012> ~14s

0:014> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0b83ed04 79ed98fd ntdll!KiFastSystemCallRet
0b83ed6c 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
0b83ed8c 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait+0x3c
0b83ee10 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
0b83ee60 79ed9a62 mscorwks!Thread::DoAppropriateWait+0x40
0b83eebc 79e78944 mscorwks!CLREvent::WaitEx+0xf7
0b83eed0 79ed7b37 mscorwks!CLREvent::Wait+0x17
0b83ef5c 79ed7a9e mscorwks!AwareLock::EnterEpilog+0x8c
0b83ef78 79ebd7e4 mscorwks!AwareLock::Enter+0x61
0b83f018 074c5681 mscorwks!JIT_MonEnterWorker_Portable+0xb3
0b83f01c 793b0d1f 0×74c5681
0b83f024 79373ecd mscorlib_ni+0×2f0d1f
0b83f038 793b0c68 mscorlib_ni+0×2b3ecd
0b83f050 79e7c74b mscorlib_ni+0×2f0c68
0b83f060 79e7c6cc mscorwks!CallDescrWorker+0×33
0b83f0e0 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
0b83f220 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
0b83f23c 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
0b83f250 79fc58cd mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
0b83f438 79ef3207 mscorwks!ThreadNative::KickOffThread_Worker+0×190
0b83f44c 79ef31a3 mscorwks!Thread::DoADCallBack+0×32a
0b83f4e0 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0b83f51c 79f01723 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0b83f52c 79f02a5d mscorwks!Thread::RaiseCrossContextException+0×434
0b83f5dc 79f02ab7 mscorwks!Thread::DoADCallBack+0xda
0b83f5f8 79ef31a3 mscorwks!Thread::DoADCallBack+0×310
0b83f68c 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0b83f6c8 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0b83f6f0 79fc57b1 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
0b83f708 79fc56ac mscorwks!ManagedThreadBase::KickOff+0×13
0b83f7a4 79f95a2e mscorwks!ThreadNative::KickOffThread+0×269
0b83ff3c 76573833 mscorwks!Thread::intermediateThreadProc+0×49
0b83ff48 77c1a9bd kernel32!BaseThreadInitThunk+0xe
0b83ff88 00000000 ntdll!LdrInitializeThunk+0×4d

0:014> ub 074c5681
074c565c 080c54          or      byte ptr [esp+edx*2],cl
074c565f 07              pop     es
074c5660 8b0d88dc7b02    mov     ecx,dword ptr ds:[27BDC88h]
074c5666 e8de009b72      call    mscorwks!JIT_MonEnterWorker (79e75749)
074c566b a1240a5407      mov     eax,dword ptr ds:[07540A24h]
074c5670 3105280a5407    xor     dword ptr ds:[7540A28h],eax
074c5676 8b0d84dc7b02    mov     ecx,dword ptr ds:[27BDC84h]
074c567c e8c8009b72      call    mscorwks!JIT_MonEnterWorker (79e75749)

0:014> dp 27BDC84h l1
027bdc84  0a1ffd4c

0:014> !dlk
Examining SyncBlocks...
Scanning for ReaderWriterLock instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for ReaderWriterLockSlim instances...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Could not find symbol ntdll!RtlCriticalSectionList.
Scanning for threads waiting on SyncBlocks...
Scanning for threads waiting on ReaderWriterLock locks...
Scanning for threads waiting on ReaderWriterLocksSlim locks...
Scanning for threads waiting on CriticalSections...
*DEADLOCK DETECTED*
CLR thread 0xd holds the lock on SyncBlock 052cbd3c OBJ:0a1ffd4c[System.String] STRVAL=critical section 1
…and is waiting for the lock on SyncBlock 052cbf1c OBJ:0a1ffd84[System.String] STRVAL=critical section 2
CLR thread 0xb holds the lock on SyncBlock 052cbf1c OBJ:0a1ffd84[System.String] STRVAL=critical section 2
…and is waiting for the lock on SyncBlock 052cbd3c OBJ:0a1ffd4c[System.String] STRVAL=critical section 1
CLR Thread 0xd is waiting at UserQuery+ClassMain.thread_proc_1()(+0×42 IL)(+0×60 Native)
CLR Thread 0xb is waiting at UserQuery+ClassMain.thread_proc_2()(+0×19 IL)(+0×21 Native)

1 deadlock detected.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a .NET counterpart to unmanaged and native code execution residue pattern. Here we can use SOS extension !DumpStack command for call level execution residue (see Caller-n-Callee pattern example) and !DumpStackObjects (!dso) for managed object references found on a raw stack:

0:011> !DumpStackObjects
OS Thread Id: 0x8e0 (11)
ESP/REG  Object   Name
09efe4b8 0a2571bc System.Threading.Thread
09efe538 0a1ffddc System.Threading.Thread
09efe844 0a1ffba8 UserQuery
09efe974 0a1ffce0 System.Signature
09efea20 0a1ffd10 System.RuntimeTypeHandle[]
09efeae8 08985e14 System.Object[]    (System.Reflection.AssemblyName[])
09efeaec 0a1ffa78 System.Diagnostics.Stopwatch
09efeaf0 0a1ffa6c LINQPad.Extensibility.DataContext.QueryExecutionManager
09efeafc 0a1ffba8 UserQuery
09efeb00 0a1ffa58 System.RuntimeType
09efeb04 08995474 LINQPad.ObjectGraph.Formatters.XhtmlWriter
09efeb08 08985dfc System.Reflection.Assembly
09efeb0c 08985dc8 LINQPad.ExecutionModel.ResultData
09efeb10 08984548 LINQPad.ExecutionModel.Server
09efebdc 0a1ffbe8 System.Reflection.RuntimeMethodInfo
09efebe0 0a1fcfc4 LINQPad.ExecutionModel.ConsoleTextReader
09efebe4 0a1fcddc System.IO.StreamReader+NullStreamReader
09efebe8 0899544c System.IO.TextWriter+SyncTextWriter
09efebec 08985efc System.Reflection.AssemblyName
09efebf0 08985d4c System.String    C:\Users\Training\AppData\Local\Temp\LINQPad\fcamvgpa
09efec30 08984548 LINQPad.ExecutionModel.Server
09efeedc 08985910 System.Threading.ThreadStart

0:011> !DumpObj 0a2571bc
Name: System.Threading.Thread
MethodTable: 790fe704
EEClass: 790fe694
Size: 56(0×38) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT    Field   Offset                 Type VT     Attr    Value Name
7910a5c4  4000634        4 ….Contexts.Context  0 instance 08980ee4 m_Context
79104de8  4000635        8 ….ExecutionContext  0 instance 00000000 m_ExecutionContext
790fd8c4  4000636        c        System.String  0 instance 00000000 m_Name
790fe3b0  4000637       10      System.Delegate  0 instance 00000000 m_Delegate
79130084  4000638       14    System.Object[][]  0 instance 00000000 m_ThreadStaticsBuckets
7912d7c0  4000639       18       System.Int32[]  0 instance 00000000 m_ThreadStaticsBits
791028f4  400063a       1c …ation.CultureInfo  0 instance 00000000 m_CurrentCulture
791028f4  400063b       20 …ation.CultureInfo  0 instance 00000000 m_CurrentUICulture
790fd0f0  400063c       24        System.Object  0 instance 00000000 m_ThreadStartArg
791016bc  400063d       28        System.IntPtr  1 instance  8f69280 DONT_USE_InternalThread
79102290  400063e       2c         System.Int32  1 instance        2 m_Priority
79102290  400063f       30         System.Int32  1 instance       11 m_ManagedThreadId
7910a7a8  4000640      168 …LocalDataStoreMgr  0   shared   static s_LocalDataStoreMgr
>> Domain:Value  000710a8:06c42ef4 08e65d48:00000000 <<
790fd0f0  4000641      16c        System.Object  0   shared   static s_SyncObject
>> Domain:Value  000710a8:017b25d8 08e65d48:0898381c <<

Although unmanaged, CLR and JIT-code residue is useful for analysis, for example, as shown in Handled Exception pattern examples.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Similar to unmanaged user space handled exceptions residue we can see similar one on raw stacks of .NET CLR threads. Here are some typical fragments (x86, CLR 4 has similar residue):

[...]
09c8e1e0  79ef2dee mscorwks!ExInfo::Init+0x41
09c8e1e4  00004000
09c8e1e8  79f088cc mscorwks!`string'
09c8e1ec  79f088c2 mscorwks!ExInfo::UnwindExInfo+0x14d
09c8e1f0  08f68728
09c8e1f4  95f5b898
09c8e1f8  09c8e1a4
09c8e1fc  09c8e92c
09c8e200  7a34d0d8 mscorwks!GetManagedNameForTypeInfo+0x22b02
09c8e204  79f091ee mscorwks!COMPlusCheckForAbort+0x15
09c8e208  00000000
09c8e20c  0aada664
09c8e210  0aaabff4
09c8e214  00000000
09c8e218  09c8eeec
09c8e21c  074c1f23
09c8e220  09c8ef0c
09c8e224  79f091cb mscorwks!JIT_EndCatch+0x16
09c8e228  09c8ef0c
09c8e22c  09c8eeec
09c8e230  074c1f23
09c8e234  09c8e25c
09c8e238  0009c108
09c8e23c  09c8e460
09c8e240  09c8e5c4
09c8e244  00071d88
09c8e248  08f68728
09c8e24c  79e734c4 mscorwks!ClrFlsSetValue+0x57
09c8e250  95f5b8e4
09c8e254  0aada634
09c8e258  08f68728
09c8e25c  0aada90c
09c8e260  0aaabff4
09c8e264  00000002
09c8e268  09c8e304
09c8e26c  0aada664
09c8e270  00000000
09c8e274  09c8ef0c
09c8e278  09c8e234
09c8e27c  074c1f13
09c8e280  00000000
09c8e284  08f688a0
09c8e288  09c8e234
09c8e28c  79f00c0b mscorwks!Thread::ReturnToContext+0x4e2
09c8e290  0aada90c
09c8e294  09c8eef4
09c8e298  09c8e2bc
09c8e29c  79f08eb8 mscorwks!EEJitManager::ResumeAtJitEH+0x28
09c8e2a0  09c8e460
09c8e2a4  074c1ed8
09c8e2a8  074b41a8
09c8e2ac  00000000
09c8e2b0  08f68728
09c8e2b4  00000000
09c8e2b8  09c8e410
09c8e2bc  09c8e3c8
09c8e2c0  79f08df5 mscorwks!COMPlusUnwindCallback+0x7c3
09c8e2c4  09c8e460
09c8e2c8  074b41a8
09c8e2cc  00000000
09c8e2d0  08f68728
09c8e2d4  00000000
09c8e2d8  0009c108
09c8e2dc  09c8e410
09c8e2e0  09c8e5c4
09c8e2e4  074b41a8
09c8e2e8  09c8e3a4
09c8e2ec  79e734c4 mscorwks!ClrFlsSetValue+0x57
09c8e2f0  95f5b984
09c8e2f4  0009c128
09c8e2f8  09c8e3a4
09c8e2fc  00000000
09c8e300  00000000
09c8e304  00000002
[...]
09c8e4e4  00000000
09c8e4e8  79f09160 mscorwks!_CT??_R0H+0x34b4
09c8e4ec  ffffffff
09c8e4f0  73792e2f msvcr80!_getptd+0x6
09c8e4f4  ffffffff
09c8e4f8  737b7a78 msvcr80!__FrameUnwindToState+0xd9
09c8e4fc  737b7a5e msvcr80!__FrameUnwindToState+0xbf
09c8e500  95f5bc05
09c8e504  e06d7363
09c8e508  1fffffff
09c8e50c  19930522
09c8e510  ffffffff
09c8e514  ffffffff
09c8e518  09c8e500
09c8e51c  09c8e554
09c8e520  09c8e5a8
09c8e524  73798cd9 msvcr80!_except_handler4
09c8e528  efbc0d3d
09c8e52c  fffffffe
09c8e530  737b7a5e msvcr80!__FrameUnwindToState+0xbf
09c8e534  737b89cb msvcr80!__InternalCxxFrameHandler+0x6d
09c8e538  09c8eab0
09c8e53c  09c8e6a0
09c8e540  79f09160 mscorwks!_CT??_R0H+0x34b4
09c8e544  ffffffff
09c8e548  00000000
09c8e54c  00000000
09c8e550  00000000
09c8e554  09c8e590
09c8e558  737b8af1 msvcr80!__CxxFrameHandler3+0x26
09c8e55c  09c8e600
09c8e560  09c8eab0
09c8e564  01010101
09c8e568  09000000
09c8e56c  09c8f160
09c8e570  07540c00
09c8e574  00071d88
09c8e578  08e65d48
09c8e57c  09c8e5ec
09c8e580  074c1ec8
09c8e584  00000024
09c8e588  00000001
09c8e58c  0009c108
09c8e590  08f68728
09c8e594  00000000
09c8e598  00000000
09c8e59c  09c8eb38
09c8e5a0  00000000
09c8e5a4  09c8e6a0
09c8e5a8  09c8f15c
09c8e5ac  09c8f15c
09c8e5b0  09c8eb38
09c8e5b4  95f5bf28
09c8e5b8  09c8e8f4
09c8e5bc  79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e5c0  08f68728
09c8e5c4  09c8ea40
09c8e5c8  79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e5cc  09c8e5ec
09c8e5d0  79f07d64 mscorwks!COMPlusUnwindCallback
09c8e5d4  09c8ea40
09c8e5d8  00000005
09c8e5dc  00000000
09c8e5e0  08f68728
09c8e5e4  08f688a0
09c8e5e8  08f68728
09c8e5ec  09c8ec20
09c8e5f0  00000000
09c8e5f4  09c8ecbc
09c8e5f8  09c8ecc0
09c8e5fc  09c8ecc4
09c8e600  09c8ecc8
09c8e604  09c8eccc
09c8e608  09c8ecd0
09c8e60c  09c8ecd4
09c8e610  09c8eeec
09c8e614  09c8ecd8
09c8e618  09c8ecd8
09c8e61c  00000024
09c8e620  00000000
09c8e624  0009c108
09c8e628  08f68728
09c8e62c  00000000
09c8e630  00000000
09c8e634  79e71ba4 mscorwks!Thread::CatchAtSafePoint
09c8e638  00000000
09c8e63c  79e71ba4 mscorwks!Thread::CatchAtSafePoint
09c8e640  09c8f15c
09c8e644  09c8f15c
09c8e648  00000000
09c8e64c  95f5bcc0
09c8e650  09c8e988
09c8e654  79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e658  09c8ea40
09c8e65c  79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e660  09c8e680
09c8e664  79f07957 mscorwks!COMPlusThrowCallback
09c8e668  09c8ea40
09c8e66c  00000000
09c8e670  00000000
09c8e674  0aada90c
09c8e678  09c8ea40
09c8e67c  79e84bff mscorwks!Thread::StackWalkFrames+0xc5
09c8e680  09c8ec20
09c8e684  00000000
09c8e688  09c8ecbc
09c8e68c  09c8ecc0
09c8e690  09c8ecc4
09c8e694  09c8ecc8
[...]
09c8e8f0  95f5b264
09c8e8f4  09c8e914
09c8e8f8  79f07d5e mscorwks!UnwindFrames+0x62
09c8e8fc  79f07d64 mscorwks!COMPlusUnwindCallback
09c8e900  09c8ea40
09c8e904  00000005
09c8e908  00000000
09c8e90c  09c8ef6c
09c8e910  08f68728
09c8e914  09c8e9a4
09c8e918  79f089cc mscorwks!COMPlusAfterUnwind+0x97
09c8e91c  08f68728
09c8e920  09c8ea40
09c8e924  00000001
09c8e928  00000000
09c8e92c  09c8ef6c
09c8e930  79f0a3d9 mscorwks!COMPlusNestedExceptionHandler
09c8e934  09c8f160
09c8e938  00000000
09c8e93c  00000000
09c8e940  cccccccc
[...]

Sometimes we can see ‘ExecuteHandler’ calls if they were not overwritten:

[...]
09d2e6e0  00000000
09d2e6e4  00000720
09d2e6e8  77c41039 ntdll!ExecuteHandler2+0x26
09d2e6ec  09d2e7c8
09d2e6f0  09d2eb60
09d2e6f4  09d2e7e4
09d2e6f8  09d2e7a4
09d2e6fc  09d2eb60
09d2e700  77c4104d ntdll!ExecuteHandler2+0x3a
09d2e704  09d2eb60
09d2e708  09d2e7b0
09d2e70c  77c4100b ntdll!ExecuteHandler+0x24
09d2e710  09d2e7c8
09d2e714  00000001
09d2e718  09d2e6b0
09d2e71c  09d2e7a4
09d2e720  09d2e784
09d2e724  76545ac9 kernel32!_except_handler4
[...]

If there are such traces they can be visible as Caller-n-Callee pattern:

0:011> !DumpStack
OS Thread Id: 0x3cc (11)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr  Caller, Callee
09d2e690 77c40690 ntdll!ZwWaitForMultipleObjects+0xc
09d2e694 76577e09 kernel32!WaitForMultipleObjectsEx+0x11d, calling ntdll!NtWaitForMultipleObjects
09d2e6d8 76578101 kernel32!WaitForMultipleObjectsEx+0x33, calling ntdll!RtlActivateActivationContextUnsafeFast
09d2e6e4 77c41039 ntdll!ExecuteHandler2+0×26
09d2e708 77c4100b ntdll!ExecuteHandler+0×24, calling ntdll!ExecuteHandler2
09d2e730 6baa516a clr!WaitForMultipleObjectsEx_SO_TOLERANT+0×56, calling kernel32!WaitForMultipleObjectsEx
09d2e794 6baa4f98 clr!Thread::DoAppropriateAptStateWait+0×4d, calling clr!WaitForMultipleObjectsEx_SO_TOLERANT
09d2e7b4 6baa4dd8 clr!Thread::DoAppropriateWaitWorker+0×17d, calling clr!Thread::DoAppropriateAptStateWait
09d2e848 6baa4e99 clr!Thread::DoAppropriateWait+0×60, calling clr!Thread::DoAppropriateWaitWorker
09d2e8b4 6baa4f17 clr!CLREvent::WaitEx+0×106, calling clr!Thread::DoAppropriateWait
09d2e8e0 6baa484b clr!CLRGetTickCount64+0×6b, calling clr!_allmul
09d2e908 6ba4d409 clr!CLREvent::Wait+0×19, calling clr!CLREvent::WaitEx
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If we don’t see exception codes from hidden exceptions when we inspect raw stack data we, nevertheless, in some cases might see execution residue left after calling exception handlers. For example, we can see that when we launch TestWER tool and select ‘Handled Exception’ checkbox:

If we then click on a button and then save a process memory dump using Task Manager we find the following traces on a raw stack:

0:000> !teb
TEB at 7efdd000
ExceptionList:        0018fe20
StackBase:            00190000
StackLimit:           0018d000
SubSystemTib:         00000000
FiberData:            00001e00
ArbitraryUserPointer: 00000000
Self:                 7efdd000
EnvironmentPointer:   00000000
ClientId:             00000b38 . 00000f98
RpcHandle:            00000000
Tls Storage:          7efdd02c
PEB Address:          7efde000
LastErrorValue:       0
LastStatusValue:      c0000034
Count Owned Locks:    0
HardErrorMode:        0

0:000> dps 0018d000 00190000
[...]
0018f414  00000000
0018f418  0018f840
0018f41c  0018f4cc
0018f420  77726a9b ntdll!ExecuteHandler+0×24
0018f424  0018f4e4
0018f428  0018f840
0018f42c  0018f534
0018f430  0018f4b8
0018f434  00412600 TestWER!_except_handler4
0018f438  00000001
0018f43c  00000000
[…]

0:000> ub 77726a9b
ntdll!ExecuteHandler+0x7:
77726a7e 33f6            xor     esi,esi
77726a80 33ff            xor     edi,edi
77726a82 ff742420        push    dword ptr [esp+20h]
77726a86 ff742420        push    dword ptr [esp+20h]
77726a8a ff742420        push    dword ptr [esp+20h]
77726a8e ff742420        push    dword ptr [esp+20h]
77726a92 ff742420        push    dword ptr [esp+20h]
77726a96 e808000000      call    ntdll!ExecuteHandler2 (77726aa3)

If we compare the output above with the raw stack fragment from second chance exception memory dump (after we relaunch TestWER, don’t select ‘Handled Exception’ checkbox and click on the big lightning button) we would see the similar call fragment:

[...]
0018f3f4  00dd0aa7
0018f3f8  0018f41c
0018f3fc  77726ac9 ntdll!ExecuteHandler2+0x26
0018f400  fffffffe
0018f404  0018ffc4
0018f408  0018f534
0018f40c  0018f4b8
0018f410  0018f840
0018f414  77726add ntdll!ExecuteHandler2+0x3a
0018f418  0018ffc4
0018f41c  0018f4cc
0018f420  77726a9b ntdll!ExecuteHandler+0×24
0018f424  0018f4e4
0018f428  0018ffc4
0018f42c  0018f534
0018f430  0018f4b8
0018f434  77750ae5 ntdll!_except_handler4
0018f438  00000000
0018f43c  0018f4e4
0018f440  0018ffc4
0018f444  77726a3d ntdll!RtlDispatchException+0×127
0018f448  0018f4e4
0018f44c  0018ffc4
0018f450  0018f534
0018f454  0018f4b8
0018f458  77750ae5 ntdll!_except_handler4
0018f45c  00000111
0018f460  0018f4e4
[…]

Sometimes, we can also see “Unwind”, “StackWalk”, “WalkFrames”, ”EH”, “Catch” functions too. Sometimes we don’t see anything because such residue was overwritten by subsequent function calls after handled exceptions happened some time in the past.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -