Crash Dump Analysis Patterns (Part 95)

In cases where we don’t see managed code exceptions or managed stack traces by default, we need to identify CLR threads in order to try various SOS commands and start digging into a managed realm. These threads are easily distinguished by mscorwks module on their stack traces (don’t forget to list full stack traces):

0:000> ~*kL 100

.  0  Id: 658.4ec Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr 
0007fc98 7c827d19 ntdll!KiFastSystemCallRet
0007fc9c 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0007fd44 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
0007fda0 6c296601 user32!RealMsgWaitForMultipleObjectsEx+0x141
0007fdc0 6c29684b duser!CoreSC::Wait+0x3a
0007fdf4 6c29693d duser!CoreSC::xwProcessNL+0xab
0007fe14 773b0c02 duser!MphProcessMessage+0x2e
0007fe5c 7c828556 user32!__ClientGetMessageMPH+0x30
0007fe84 7739c811 ntdll!KiUserCallbackDispatcher+0x2e
0007fea4 7f072fd6 user32!NtUserGetMessage+0xc
0007fec0 010080ef mfc42u!CWinThread::PumpMessage+0x16
0007fef0 7f072dda mmc!CAMCApp::PumpMessage+0x37
0007ff08 7f044d5b mfc42u!CWinThread::Run+0x4a
0007ff1c 01034e19 mfc42u!AfxWinMain+0x7b
0007ffc0 77e6f23b mmc!wWinMainCRTStartup+0x19d
0007fff0 00000000 kernel32!BaseProcessStart+0x23

   1  Id: 658.82c Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr 
003afea0 7c827d19 ntdll!KiFastSystemCallRet
003afea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
003aff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
003affb8 77e6482f ntdll!EtwpEventPump+0x27f
003affec 00000000 kernel32!BaseThreadStart+0x34

   2  Id: 658.648 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr 
00f3fe18 7c827859 ntdll!KiFastSystemCallRet
00f3fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
00f3ff84 77c88792 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
00f3ff8c 77c8872d rpcrt4!RecvLotsaCallsWrapper+0xd
00f3ffac 77c7b110 rpcrt4!BaseCachedThreadRoutine+0x9d
00f3ffb8 77e6482f rpcrt4!ThreadStartRoutine+0x1b
00f3ffec 00000000 kernel32!BaseThreadStart+0x34

   3  Id: 658.640 Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr 
0156fdb4 7c827d19 ntdll!KiFastSystemCallRet
0156fdb8 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0156fe60 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
0156febc 6c296601 user32!RealMsgWaitForMultipleObjectsEx+0x141
0156fedc 6c29684b duser!CoreSC::Wait+0x3a
0156ff10 6c28f9e6 duser!CoreSC::xwProcessNL+0xab
0156ff30 6c28bce1 duser!GetMessageExA+0x44
0156ff84 77bcb530 duser!ResourceManager::SharedThreadProc+0xb6
0156ffb8 77e6482f msvcrt!_endthreadex+0xa3
0156ffec 00000000 kernel32!BaseThreadStart+0x34

   4  Id: 658.e74 Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr 
01d1fe30 7c827d19 ntdll!KiFastSystemCallRet
01d1fe34 77e6202c ntdll!NtWaitForMultipleObjects+0xc
01d1fedc 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
01d1fef8 79f02541 kernel32!WaitForMultipleObjects+0x18
01d1ff58 79f0249e mscorwks!DebuggerRCThread::MainLoop+0xe9
01d1ff88 79f023c5 mscorwks!DebuggerRCThread::ThreadProc+0xe5
01d1ffb8 77e6482f mscorwks!DebuggerRCThread::ThreadProcStatic+0×9c

01d1ffec 00000000 kernel32!BaseThreadStart+0×34

   5  Id: 658.4d4 Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr 
03dffcc4 7c827d19 ntdll!KiFastSystemCallRet
03dffcc8 77e6202c ntdll!NtWaitForMultipleObjects+0xc
03dffd70 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
03dffd8c 79f92bcb kernel32!WaitForMultipleObjects+0x18
03dffdac 79f97028 mscorwks!WKS::WaitForFinalizerEvent+0×77
03dffdc0 79e9845f mscorwks!WKS::GCHeap::FinalizerThreadWorker+0×49
03dffdd4 79e983fb mscorwks!Thread::DoADCallBack+0×32a
03dffe68 79e98321 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
03dffea4 79eef6cc mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
03dffecc 79eef6dd mscorwks!ManagedThreadBase_NoADTransition+0×32
03dffedc 79f3c63c mscorwks!ManagedThreadBase::FinalizerBase+0xd
03dfff14 79f92015 mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb
03dfffb8 77e6482f mscorwks!Thread::intermediateThreadProc+0×49

03dfffec 00000000 kernel32!BaseThreadStart+0×34

   6  Id: 658.f54 Suspend: 1 Teb: 7ffd6000 Unfrozen
ChildEBP RetAddr 
040afec4 7c826f69 ntdll!KiFastSystemCallRet
040afec8 77e41ed5 ntdll!NtDelayExecution+0xc
040aff30 79fd8a41 kernel32!SleepEx+0x68
040affac 79fd88ef mscorwks!ThreadpoolMgr::TimerThreadFire+0×6d
040affb8 77e6482f mscorwks!ThreadpoolMgr::TimerThreadStart+0×57

040affec 00000000 kernel32!BaseThreadStart+0×34

   7  Id: 658.988 Suspend: 1 Teb: 7ffd5000 Unfrozen
ChildEBP RetAddr 
0410fc2c 7c827d29 ntdll!KiFastSystemCallRet
0410fc30 77e61d1e ntdll!ZwWaitForSingleObject+0xc
0410fca0 79e8c5f9 kernel32!WaitForSingleObjectEx+0xac
0410fce4 79e8c52f mscorwks!PEImage::LoadImage+0×1af
0410fd34 79e8c54e mscorwks!CLREvent::WaitEx+0×117
0410fd48 79ee3f35 mscorwks!CLREvent::Wait+0×17
0410fe14 79f92015 mscorwks!AppDomain::ADUnloadThreadStart+0×308
0410ffb8 77e6482f mscorwks!Thread::intermediateThreadProc+0×49

0410ffec 00000000 kernel32!BaseThreadStart+0×34

   8  Id: 658.e0 Suspend: 1 Teb: 7ff4f000 Unfrozen
ChildEBP RetAddr 
0422fcec 7c827d19 ntdll!KiFastSystemCallRet
0422fcf0 7c83c7be ntdll!NtWaitForMultipleObjects+0xc
0422ffb8 77e6482f ntdll!RtlpWaitThread+0x161
0422ffec 00000000 kernel32!BaseThreadStart+0x34

   9  Id: 658.db4 Suspend: 1 Teb: 7ff4e000 Unfrozen
ChildEBP RetAddr 
0447fec0 7c827d19 ntdll!KiFastSystemCallRet
0447fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0447ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
0447ff88 76929e35 kernel32!WaitForMultipleObjects+0x18
0447ffb8 77e6482f userenv!NotificationThread+0x5f
0447ffec 00000000 kernel32!BaseThreadStart+0x34

  10  Id: 658.e7c Suspend: 1 Teb: 7ff4c000 Unfrozen
ChildEBP RetAddr 
0550ff7c 7c8277f9 ntdll!KiFastSystemCallRet
0550ff80 71b25914 ntdll!NtRemoveIoCompletion+0xc
0550ffb8 77e6482f mswsock!SockAsyncThread+0x69
0550ffec 00000000 kernel32!BaseThreadStart+0x34

[...]

- Dmitry Vostokov @ DumpAnalysis.org -

3 Responses to “Crash Dump Analysis Patterns (Part 95)”

  1. Crash Dump Analysis » Blog Archive » 10 Common Mistakes in Memory Analysis (Part 6) Says:

    […] previous thread #16 is a CLR thread loading an […]

  2. Marc Sherman Says:

    Thanks for the cool idea for checking if an app is managed (or at least a subset of it).

    For silverlight apps you’d want to look for coreclr instead of mscorwks in the stack traces. See http://blogs.msdn.com/tess/archive/2008/08/21/debugging-silverlight-applications-with-windbg-and-sos-dll.aspx for the details.

    I’m looking forward to your managed articles since I too am a native programmer delving into the .NET world.

    Marc

  3. Dmitry Vostokov Says:

    Thanks! I’ve found that in CLR 4.0 the module has changed to just clr:

    0:000> lmv m clr
    start             end                 module name
    000007fe`eadc0000 000007fe`eb725000   clr        (pdb symbols)
        Loaded symbol image file: clr.dll
        Image path: C:\Windows\Microsoft.NET\Framework64\
    v4.0.30319\clr.dll
        Image name: clr.dll
        Timestamp:        Thu Mar 18 12:39:07 2010 (4BA21EEB)
        CheckSum:         00959DBD
        ImageSize:        00965000
        File version:     4.0.30319.1
        Product version:  4.0.30319.1
        File flags:       8 (Mask 3F) Private
        File OS:          4 Unknown Win32
        File type:        2.0 Dll
        File date:        00000000.00000000
        Translations:     0409.04b0
        CompanyName:      Microsoft Corporation
        ProductName:      Microsoft® .NET Framework
        InternalName:     clr.dll
        OriginalFilename: clr.dll
        ProductVersion:   4.0.30319.1
        FileVersion:      4.0.30319.1 (RTMRel.030319-0100)
        PrivateBuild:     DDBLD431
        FileDescription:  Microsoft .NET Runtime Common Language
    Runtime - WorkStation
        LegalCopyright:   © Microsoft Corporation.
    All rights reserved.
        Comments:         Flavor=Retail

    0:000> ~16kc
    Call Site
    ntdll!NtWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    clr!CLREvent::WaitEx
    clr!CLREvent::WaitEx
    clr!CLREvent::WaitEx
    clr!Thread::WaitSuspendEventsHelper
    clr!Thread::WaitSuspendEvents
    clr! ?? ::FNODOBFM::`string'
    clr!Thread::RareDisablePreemptiveGC
    clr!GCHolderBase<1,0,0,1>::EnterInternal
    clr!AddTimerCallbackEx
    clr!ThreadpoolMgr::AsyncTimerCallbackCompletion
    clr!UnManagedPerAppDomainTPCount::DispatchWorkItem
    clr!ThreadpoolMgr::NewWorkerThreadStart
    clr!ThreadpoolMgr::WorkerThreadStart
    clr!Thread::intermediateThreadProc
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

Leave a Reply