Crash Dump Analysis Patterns (Part 17)
.NET programs also crash either from defects in .NET runtime (Common Language Runtime, CLR) or from non-handled runtime exceptions in managed code executed by .NET virtual machine. The latter exceptions are re-thrown from .NET runtime to be handled by operating system and intercepted by native debuggers. Therefore our next crash dump analysis pattern is called Managed Code Exception.
When you get a dump from .NET application it is the dump from a native process. !analyze -v output can usually tell you that exception is actually CLR exception and give you other hints to look at managed code stack (CLR stack):
FAULTING_IP:
kernel32!RaiseException+53
77e4bee7 5e pop esi
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 77e4bee7 (kernel32!RaiseException+0x00000053)
ExceptionCode: e0434f4d (CLR exception)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 80131604
DEFAULT_BUCKET_ID: CLR_EXCEPTION
PROCESS_NAME: mmc.exe
ERROR_CODE: (NTSTATUS) 0xe0434f4d - <Unable to get error code text>
MANAGED_STACK: !dumpstack -EE
No export dumpstack found
STACK_TEXT:
05faf3d8 79f97065 e0434f4d 00000001 00000001 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
05faf438 7a0945a4 023f31e0 00000000 00000000 mscorwks!DllCanUnloadNowInternal+0×37a9
05faf4fc 00f2f00a 02066be4 02085ee8 023d0df0 mscorwks!CorLaunchApplication+0×12005
05faf500 02066be4 02085ee8 023d0df0 023d0e2c 0xf2f00a
05faf504 02085ee8 023d0df0 023d0e2c 05e00dfa 0×2066be4
05faf508 023d0df0 023d0e2c 05e00dfa 023d0e10 0×2085ee8
05faf50c 023d0e2c 05e00dfa 023d0e10 05351d30 0×23d0df0
05faf510 05e00dfa 023d0e10 05351d30 023d0e10 0×23d0e2c
FOLLOWUP_IP:
mscorwks!DllCanUnloadNowInternal+37a9
79f97065 c745fcfeffffff mov dword ptr [ebp-4],0FFFFFFFEh
SYMBOL_NAME: mscorwks!DllCanUnloadNowInternal+37a9
MODULE_NAME: mscorwks
IMAGE_NAME: mscorwks.dll
PRIMARY_PROBLEM_CLASS: CLR_EXCEPTION
BUGCHECK_STR: APPLICATION_FAULT_CLR_EXCEPTION
Sometimes you can see mscorwks.dll on raw stack or see it loaded and find it on other thread stacks than the current one.
When you get such hints you might want to get managed code stack as well. First you need to load the appropriate WinDbg SOS extension (Son of Strike) corresponding to .NET runtime version. This can be done by the following command:
0:015> .loadby sos mscorwks
You can check which SOS extension version was loaded this by using .chain command:
0:015> .chain
Extension DLL search Path:
...
...
...
Extension DLL chain:
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos: image 2.0.50727.42, API 1.0.0, built Fri Sep 23 08:27:26 2005
[path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll]
dbghelp: image 6.6.0007.5, API 6.0.6, built Sat Jul 08 21:11:32 2006
[path: C:\Program Files\Debugging Tools for Windows\dbghelp.dll]
ext: image 6.6.0007.5, API 1.0.0, built Sat Jul 08 21:10:52 2006
[path: C:\Program Files\Debugging Tools for Windows\winext\ext.dll]
exts: image 6.6.0007.5, API 1.0.0, built Sat Jul 08 21:10:48 2006
[path: C:\Program Files\Debugging Tools for Windows\WINXP\exts.dll]
uext: image 6.6.0007.5, API 1.0.0, built Sat Jul 08 21:11:02 2006
[path: C:\Program Files\Debugging Tools for Windows\winext\uext.dll]
ntsdexts: image 6.0.5457.0, API 1.0.0, built Sat Jul 08 21:29:38 2006
[path: C:\Program Files\Debugging Tools for Windows\WINXP\ntsdexts.dll]
Then you can use !dumpstack to dump the current stack or !EEStack command to dump all thread stacks. The native stack trace would be mixed with managed stack trace:
0:015> !dumpstack
OS Thread Id: 0x16e8 (15)
Current frame: kernel32!RaiseException+0x53
ChildEBP RetAddr Caller,Callee
05faf390 77e4bee7 kernel32!RaiseException+0x53, calling ntdll!RtlRaiseException
05faf3a8 79e814da mscorwks!Binder::RawGetClass+0x23, calling mscorwks!Module::LookupTypeDef
05faf3bc 79e87ff4 mscorwks!Binder::IsClass+0x21, calling mscorwks!Binder::RawGetClass
05faf3c8 79f958b8 mscorwks!Binder::IsException+0x13, calling mscorwks!Binder::IsClass
05faf3d8 79f97065 mscorwks!RaiseTheExceptionInternalOnly+0x226, calling kernel32!RaiseException
05faf438 7a0945a4 mscorwks!JIT_Throw+0xd0, calling mscorwks!RaiseTheExceptionInternalOnly
05faf4ac 7a0944ea mscorwks!JIT_Throw+0x1e, calling mscorwks!LazyMachStateCaptureState
05faf4c8 793d424e (MethodDesc 0x7924ad68 +0x2e System.Threading.WaitHandle.WaitOne(Int64, Boolean)), calling mscorwks!WaitHandleNative::CorWaitOneNative
05faf4fc 00f2f00a (MethodDesc 0x4f97500 +0x9a Ironring.Management.MMC.SnapinBase+MmcWindow.Invoke(System.Delegate, System.Object[])), calling mscorwks!JIT_Throw
05faf510 05e00dfa (MethodDesc 0×4f98fd8 +0xca MyNamespace.MyClass.MyMethod(Boolean)), calling 05fc7124
05faf55c 00f62fbc (MethodDesc 0×4f95f90 +0×16f4 MyNamespace.MyClass.MyMethod.Initialise(System.Object))
05faf740 793d912f (MethodDesc 0×7925fc70 +0×2f System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object))
05faf748 793683dd (MethodDesc 0×7913f3d0 +0×81 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
05faf75c 793d9218 (MethodDesc 0×7925fc80 +0×6c System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)), calling (MethodDesc 0×7913f3d0 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
05faf774 79e88f63 mscorwks!CallDescrWorker+0×33
05faf784 79e88ee4 mscorwks!CallDescrWorkerWithHandler+0xa3, calling mscorwks!CallDescrWorker
05faf804 79f20212 mscorwks!DispatchCallBody+0×1e, calling mscorwks!CallDescrWorkerWithHandler
05faf824 79f201bc mscorwks!DispatchCallDebuggerWrapper+0×3d, calling mscorwks!DispatchCallBody
05faf888 79f2024b mscorwks!DispatchCallNoEH+0×51, calling mscorwks!DispatchCallDebuggerWrapper
05faf8bc 7a07bdf0 mscorwks!Holder
05faf90c 77e61d1e kernel32!WaitForSingleObjectEx+0xac, calling ntdll!ZwWaitForSingleObject
05faf91c 79ecb4a4 mscorwks!Thread::UserResumeThread+0xfb
05faf92c 79ecb442 mscorwks!Thread::DoADCallBack+0×355, calling mscorwks!Thread::UserResumeThread+0xae
05faf950 79e74afe mscorwks!Thread::EnterRuntimeNoThrow+0×9b, calling mscorwks!_EH_epilog3
05faf988 79e77fe8 mscorwks!PEImage::LoadImage+0×1e1, calling mscorwks!_SEH_epilog4
05faf9c0 79ecb364 mscorwks!Thread::DoADCallBack+0×541, calling mscorwks!Thread::DoADCallBack+0×2a5
05faf9fc 7a0e1b7e mscorwks!Thread::DoADCallBack+0×575, calling mscorwks!Thread::DoADCallBack+0×4d4
05fafa24 7a0e1bab mscorwks!ManagedThreadBase::ThreadPool+0×13, calling mscorwks!Thread::DoADCallBack+0×550
05fafa38 7a07cae8 mscorwks!QueueUserWorkItemCallback+0×9d, calling mscorwks!ManagedThreadBase::ThreadPool
05fafa54 7a07ca48 mscorwks!QueueUserWorkItemCallback, calling mscorwks!UnwindAndContinueRethrowHelperAfterCatch
05fafa90 7a110f08 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0×40
05fafaa8 7a112328 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×1f2, calling mscorwks!ThreadpoolMgr::ExecuteWorkRequest
05fafad0 79e7839d mscorwks!EEHeapFreeInProcessHeap+0×21, calling mscorwks!EEHeapFree
05fafae0 79e782dc mscorwks!operator delete[]+0×30, calling mscorwks!EEHeapFreeInProcessHeap
05fafb14 79ecb00b mscorwks!Thread::intermediateThreadProc+0×49
05fafb48 77e65512 kernel32!FlsSetValue+0xc7, calling kernel32!_SEH_epilog
05fafb6c 75da14d0 sxs!_calloc_crt+0×19, calling sxs!calloc
05fafb80 77e65512 kernel32!FlsSetValue+0xc7, calling kernel32!_SEH_epilog
05fafb88 75da1401 sxs!_CRT_INIT+0×17e, calling sxs!_initptd
05fafb8c 75da1408 sxs!_CRT_INIT+0×185, calling kernel32!GetCurrentThreadId
05fafb9c 30403805 MMCFormsShim!DllMain+0×15, calling MMCFormsShim!PrxDllMain
05fafbb0 30418b69 MMCFormsShim!__DllMainCRTStartup+0×7a, calling MMCFormsShim!DllMain
05fafbdc 75de0e4c sxs!_SxsDllMain+0×87, calling sxs!DllStartup_CrtInit
05fafbf0 30418bf9 MMCFormsShim!__DllMainCRTStartup+0×10a, calling MMCFormsShim!__SEH_epilog4
05fafbf4 30418c22 MMCFormsShim!_DllMainCRTStartup+0×1d, calling MMCFormsShim!__DllMainCRTStartup
05fafbfc 7c81a352 ntdll!LdrpCallInitRoutine+0×14
05fafc24 7c82ee8b ntdll!LdrpInitializeThread+0×1a5, calling ntdll!RtlLeaveCriticalSection
05fafc2c 7c82edec ntdll!LdrpInitializeThread+0×18f, calling ntdll!_SEH_epilog
05fafc7c 7c82ed71 ntdll!LdrpInitializeThread+0xd8, calling ntdll!RtlActivateActivationContextUnsafeFast
05fafc80 7c82ed35 ntdll!LdrpInitializeThread+0×12c, calling ntdll!RtlDeactivateActivationContextUnsafeFast
05fafcb4 7c82edec ntdll!LdrpInitializeThread+0×18f, calling ntdll!_SEH_epilog
05fafcb8 7c827c3b ntdll!NtTestAlert+0xc
05fafcbc 7c82ecb1 ntdll!_LdrpInitialize+0×1de, calling ntdll!_SEH_epilog
05fafd10 7c82ecb1 ntdll!_LdrpInitialize+0×1de, calling ntdll!_SEH_epilog
05fafd14 7c826d9b ntdll!NtContinue+0xc
05fafd18 7c8284da ntdll!KiUserApcDispatcher+0×3a, calling ntdll!NtContinue
05faffa4 79ecaff9 mscorwks!Thread::intermediateThreadProc+0×37, calling mscorwks!_alloca_probe_16
05faffb8 77e64829 kernel32!BaseThreadStart+0×34
.NET language symbolic names are usually reconstructed from .NET assembly metadata.
You can examine a CLR exception and get managed stack trace by using !PrintException and !CLRStack commands, for example:
0:014> !PrintException
Exception object: 02320314
Exception type: System.Reflection.TargetInvocationException
Message: Exception has been thrown by the target of an invocation.
InnerException: System.Runtime.InteropServices.COMException, use !PrintException 023201a8 to see more
StackTrace (generated):
SP IP Function
075AF4FC 016BFD9A Ironring.Management.MMC.SnapinBase+MmcWindow.Invoke(System.Delegate, System.Object[])
...
...
...
075AF740 793D87AF System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
075AF748 793608FD System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
075AF760 793D8898 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
StackTraceString: <none>
HResult: 80131604
0:014> !PrintException 023201a8
Exception object: 023201a8
Exception type: System.Runtime.InteropServices.COMException
Message: Error HRESULT E_FAIL has been returned from a call to a COM component.
InnerException: <none>
StackTrace (generated):
SP IP Function
00000000 00000001 Ironring.Management.MMC.IMMCFormsShim.HostUserControl3(System.Object, System.Object, System.String, System.String, Int32, Int32)
0007F724 073875B9 Ironring.Management.MMC.FormNode.SetShimControl(System.Object)
0007F738 053D9DDE Ironring.Management.MMC.FormNode.set_ControlType(System.Type)
...
...
...
StackTraceString: <none>
HResult: 80004005
0:014> !CLRStack
OS Thread Id: 0x11ec (14)
ESP EIP
075af4fc 016bfd9a Ironring.Management.MMC.SnapinBase+MmcWindow.Invoke(System.Delegate, System.Object[])
...
...
...
075af740 793d87af System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
075af748 793608fd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
075af760 793d8898 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
075af8f0 79e7be1b [GCFrame: 075af8f0]
!help command gives the list of other available SOS extension commands:
0:014> !help
Object Inspection
DumpObj (do)
DumpArray (da)
DumpStackObjects (dso)
DumpHeap
DumpVC
GCRoot
ObjSize
FinalizeQueue
PrintException (pe)
TraverseHeap
Examining code and stacks
Threads
CLRStack
IP2MD
U
DumpStack
EEStack
GCInfo
EHInfo
COMState
BPMD
Examining CLR data structures
DumpDomain
EEHeap
Name2EE
SyncBlk
DumpMT
DumpClass
DumpMD
Token2EE
EEVersion
DumpModule
ThreadPool
DumpAssembly
DumpMethodSig
DumpRuntimeTypes
DumpSig
RCWCleanupList
DumpIL
Diagnostic Utilities
VerifyHeap
DumpLog
FindAppDomain
SaveModule
GCHandles
GCHandleLeaks
VMMap
VMStat
ProcInfo
StopOnException (soe)
MinidumpMode
Other
FAQ
If you are new to .NET and interested in .NET debugging I would recommend 3 books:
Essential .NET, Volume I: The Common Language Runtime
Debugging Microsoft .NET 2.0 Applications
Advanced .NET Debugging
Expert .NET 2.0 IL Assembler
- Dmitry Vostokov @ DumpAnalysis.org -
July 16th, 2007 at 1:32 pm
In the case where .NET CLR runtime is version 1.x you get messages pointing to some .NET DLL and this could be the indication that some threads have managed code:
*** WARNING: Unable to verify checksum for mscorlib.dll*** ERROR: Module load completed but symbols could not be loaded for mscorlib.dll
In some cases we cannot load the appropriate SOS extension automatically:
0:000> .loadby sos mscorwksUnable to find module “mscorwks”
Then try SOS version 1.0
0:000> !clr10\sos.EEStackLoaded Son of Strike data table version 5 from “C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll”
August 18th, 2007 at 9:15 pm
Actually the following message means that the server version of CLR is used:
0:000> .loadby sos mscorwks
Unable to find module “mscorwks”
0:000> .loadby sos mscorsvr
0:000> !help
SOS : Help
…
…
…
October 10th, 2007 at 5:35 pm
For some crash dumps we get the following message saying that sos.dll cannot be found:
0:000> .loadby sos mscorwks
The call to LoadLibrary(C:\WIN_NO_SP\Microsoft.NET \Framework\v2.0.50727\sos) failed, Win32 error 0n126
“The specified module could not be found.”
Please check your debugger configuration and/or network access
Here we need to check where
Microsoft.NET\Framework\v2.0.50727\sos.dll
is installed on our crash dump analysis host and use .load command:
0:000> .load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll
November 8th, 2007 at 1:08 pm
The current version of WinDbg 6.8.4.0 and !analyze -v command show both native and managed stack traces from .NET 64-bit application memory dump so there is no need to load SOS manually
October 21st, 2008 at 11:31 am
[…] simple pattern I planned to write about long time ago. It is called C++ Exception. It is similar to Managed Code Exception and can manifested by the same RaiseException call on top of the stack (red). However it is […]
June 2nd, 2009 at 6:22 pm
Very useful! Thanks.
July 15th, 2009 at 7:15 pm
[…] kernel32.dll, for example (Windows API DLL). For some examples from crash dumps see patterns: Managed Code Exception or JIT […]
October 28th, 2009 at 11:31 am
[…] we have a managed code exception that was enveloping a handled unmanaged code exception, Mixed (Nested) […]
November 24th, 2009 at 4:52 pm
[…] specific SOS extension. For example, x64 WinDbg is not able to analyze the managed stack for a managed code exception in 32-bit […]
December 7th, 2009 at 5:03 pm
[…] cases where we don’t see managed code exceptions or managed stack traces by default, we need to identify CLR threads in order to try various SOS […]
May 6th, 2010 at 4:01 pm
[…] Today we introduce an icon for Managed Code Exception pattern: […]
August 10th, 2010 at 5:29 pm
[…] .load vs. .loadby (pp. 8 - 11) - Some additional load scenarios for legacy SOS and its server version can be found in comments to Managed Code Exception pattern: http://www.dumpanalysis.org/blog/index.php/2007/07/20/crash-dump-analysis-patterns-part-17/ […]