10 Common Mistakes in Memory Analysis (Part 6)

Some debugger commands or commands they invoke can be context-sensitive and their diagnostic output can depend on a current thread or a process set in a debugger, not to mention loaded debugger extensions and even their load order. Therefore, it is advisable to be context-conscious about or at least to know about context sensitivity. For example, in one mmc.exe process memory dump a default analysis command in x64 WinDbg doesn’t show any managed stack trace reported by a user who had seen it in a failure dialog box:

0:000> !analyze -v

[...]

MANAGED_BITNESS_MISMATCH:
Managed code needs matching platform of sos.dll for proper analysis. Use 'x86' debugger.

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT

STACK_TEXT: 
0007fc98 7c827d19 77e6202c 00000002 0007fce8 ntdll!KiFastSystemCallRet
0007fc9c 77e6202c 00000002 0007fce8 00000001 ntdll!NtWaitForMultipleObjects+0xc
0007fd44 7739bbd1 00000002 0007fd6c 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
0007fda0 6c296601 00000001 0007fdd4 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
0007fdc0 6c29684b 000004ff ffffffff 00000001 duser!CoreSC::Wait+0x3a
0007fdf4 6c29693d 0007fe34 00000000 00000000 duser!CoreSC::xwProcessNL+0xab
0007fe14 773b0c02 0007fe34 00000000 00000000 duser!MphProcessMessage+0x2e
0007fe5c 7c828556 0007fe74 00000014 0007ffb0 user32!__ClientGetMessageMPH+0x30
0007fe84 7739c811 7739c844 01116894 00000000 ntdll!KiUserCallbackDispatcher+0x2e
0007fea4 7f072fd6 01116894 00000000 00000000 user32!NtUserGetMessage+0xc
0007fec0 010080ef 01116894 01116860 00000002 mfc42u!CWinThread::PumpMessage+0x16
0007fef0 7f072dda 01116860 01116860 ffffffff mmc!CAMCApp::PumpMessage+0x37
0007ff08 7f044d5b ffffffff 00000002 7ffd9000 mfc42u!CWinThread::Run+0x4a
0007ff1c 01034e19 01000000 00000000 00020710 mfc42u!AfxWinMain+0x7b
0007ffc0 77e6f23b 00000000 00000000 7ffd9000 mmc!wWinMainCRTStartup+0x19d
0007fff0 00000000 01034cb0 00000000 78746341 kernel32!BaseProcessStart+0x23

Instead of concluding that the dump file wasn’t saved at the time of the failure we pay attention to all aspects of the default analysis and see that we need a platform-specific debugger. We load the same dump file into x86 WinDbg: 

0:000> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

Now we see that we need to load SOS extension explicitly and retry: 

0:000> .load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll

0:000> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x4ec (0)
Current frame:
ChildEBP RetAddr  Caller,Callee

Managed stack trace is empty here but look at all threads (we list full traces in order not to miss any module) we find one that shows a dialog box reporting a failure:

0:000> ~*kL 100

[...]

  17  Id: 658.7e4 Suspend: 1 Teb: 7ff48000 Unfrozen
ChildEBP RetAddr 
06b4f498 7739bf53 ntdll!KiFastSystemCallRet
06b4f4d0 7738965e user32!NtUserWaitMessage+0xc
06b4f4f8 773896a0 user32!InternalDialogBox+0xd0
06b4f518 773896e8 user32!DialogBoxIndirectParamAorW+0x37
06b4f53c 4afde2e1 user32!DialogBoxParamW+0×3f
06b4f584 4b05c4bc mmcndmgr!IsolationAwareDialogBoxParamW+0×4e
06b4f5a4 4b05c6eb mmcndmgr!ATL::CDialogImpl<CSnapInFailureReportDialog, ATL::CWindow>::DoModal+0×4f
06b4f68c 77c80193 mmcndmgr!CSnapInFailureReporter::ReportSnapInFailure+0×195

06b4f6b8 77ce33e1 rpcrt4!Invoke+0×30
06b4fab8 77ce2ed5 rpcrt4!NdrStubCall2+0×299
06b4fb10 7778d01b rpcrt4!CStdStubBuffer_Invoke+0xc6
06b4fb54 7778cfc8 ole32!SyncStubInvoke+0×37
06b4fb9c 776c120b ole32!StubInvoke+0xa7
06b4fc78 776c0bf5 ole32!CCtxComChnl::ContextInvoke+0xec
06b4fc94 776bc455 ole32!MTAInvoke+0×1a
06b4fcc0 7778ced5 ole32!STAInvoke+0×48
06b4fcf4 7778cd66 ole32!AppInvoke+0xa3
06b4fdc8 7778c24d ole32!ComInvokeWithLockAndIPID+0×2c5
06b4fdf0 776bc344 ole32!ComInvoke+0xca
06b4fe04 776bc30f ole32!ThreadDispatch+0×23
06b4fe1c 7739b6e3 ole32!ThreadWndProc+0xfe
06b4fe48 7739b874 user32!InternalCallWinProc+0×28
06b4fec0 7739ba92 user32!UserCallWinProcCheckWow+0×151
06b4ff28 7739bad0 user32!DispatchMessageWorker+0×327
06b4ff38 7768ffdc user32!DispatchMessageW+0xf
06b4ff6c 7768f366 ole32!CDllHost::STAWorkerLoop+0×5c
06b4ff88 7768f2a2 ole32!CDllHost::WorkerThread+0xc8
06b4ff90 776bbab4 ole32!DLLHostThreadEntry+0xd
06b4ffac 776b1704 ole32!CRpcThread::WorkerLoop+0×26
06b4ffb8 77e6482f ole32!CRpcThreadCache::RpcWorkerThreadEntry+0×20
06b4ffec 00000000 kernel32!BaseThreadStart+0×34

The previous thread #16 is a CLR thread loading an assembly:

  16  Id: 658.28c Suspend: 1 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr 
0714d024 7c827d29 ntdll!KiFastSystemCallRet
0714d028 77e61d1e ntdll!ZwWaitForSingleObject+0xc
0714d098 73ca790b kernel32!WaitForSingleObjectEx+0xac
0714d0c4 73ca485a cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x12f
0714d0f0 73ca37ce cryptnet!CryptRetrieveObjectByUrlW+0x9b
0714d168 73ca4a60 cryptnet!RetrieveObjectByUrlValidForSubject+0x5b
0714d1b8 73ca3525 cryptnet!RetrieveTimeValidObjectByUrl+0xbc
0714d220 73ca3473 cryptnet!CTVOAgent::GetTimeValidObjectByUrl+0xc2
0714d2d0 73ca3314 cryptnet!CTVOAgent::GetTimeValidObject+0x2f1
0714d300 73ca2c00 cryptnet!FreshestCrlFromCrlGetTimeValidObject+0x2d
0714d344 73ca43a4 cryptnet!CryptGetTimeValidObject+0x58
0714d3a0 73ca3122 cryptnet!GetTimeValidCrl+0x1e0
0714d3e4 73ca3080 cryptnet!GetBaseCrl+0x34
0714d484 761d9033 cryptnet!MicrosoftCertDllVerifyRevocation+0x128
0714d514 761d8eef crypt32!I_CryptRemainingMilliseconds+0x21b
0714d584 761cf39f crypt32!CertVerifyRevocation+0xb7
0714d604 761c6966 crypt32!CChainPathObject::CalculateRevocationStatus+0x1f2
0714d64c 761c6771 crypt32!CChainPathObject::CalculateAdditionalStatus+0x147
0714d708 761c78bc crypt32!CCertChainEngine::CreateChainContextFromPathGraph+0x227
0714d738 761c783f crypt32!CCertChainEngine::GetChainContext+0x44
0714d760 76bb6d8f crypt32!CertGetCertificateChain+0x60
0714d7c4 76bb6bbc wintrust!_WalkChain+0x1a8
0714d800 76bb39ef wintrust!WintrustCertificateTrust+0xb7
0714d8f4 76bb31e2 wintrust!_VerifyTrust+0x144
0714d918 64025b1b wintrust!WinVerifyTrust+0x4e
0714d9bc 7a117c85 mscorsec!GetPublisher+0xe4
0714da14 79ebeccb mscorwks!PEFile::CheckSecurity+0xcb
0714da3c 79ebec14 mscorwks!PEAssembly::DoLoadSignatureChecks+0x3a
0714da64 79ebf05a mscorwks!PEAssembly::PEAssembly+0x109
0714dd00 79ebf155 mscorwks!PEAssembly::DoOpen+0x103
0714dd94 79eb8ff2 mscorwks!PEAssembly::Open+0x79
0714def8 79eb6a5e mscorwks!AppDomain::BindAssemblySpec+0x247
0714df90 79eb691c mscorwks!PEFile::LoadAssembly+0×95
0714e030 79eb68c0 mscorwks!Module::LoadAssembly+0xee

0714e06c 79e92873 mscorwks!Assembly::FindModuleByTypeRef+0×113
0714e0d8 79fc3dc8 mscorwks!ClassLoader::ResolveTokenToTypeDefThrowing+0×88
0714e12c 79fc953d mscorwks!CEEInfo::AddDependencyOnClassToken+0×103
0714e158 79fc61cf mscorwks!CEEInfo::ScanForModuleDependencies+0xa3
0714e1fc 7908bce1 mscorwks!CEEInfo::getArgType+0×256
0714e214 7908bc5b mscorjit!Compiler::eeGetArgType+0×23
0714e25c 79067745 mscorjit!Compiler::impInlineInitVars+0×3c3
0714e4fc 790673d5 mscorjit!Compiler::fgInvokeInlineeCompiler+0×95
0714e518 79067400 mscorjit!Compiler::fgMorphCallInline+0×41
0714e52c 79065272 mscorjit!Compiler::fgInline+0×30
0714e534 7906513e mscorjit!Compiler::fgMorph+0×45
0714e544 79065b8e mscorjit!Compiler::compCompile+0×83
0714e590 79065d33 mscorjit!Compiler::compCompile+0×44f
0714e618 79066448 mscorjit!jitNativeCode+0xef
0714e63c 79fc7198 mscorjit!CILJit::compileMethod+0×25
0714e6a8 79fc722d mscorwks!invokeCompileMethodHelper+0×72
0714e6ec 79fc72a0 mscorwks!invokeCompileMethod+0×31
0714e740 79fc7019 mscorwks!CallCompileMethodWithSEHWrapper+0×5b
0714eae8 79fc6ddb mscorwks!UnsafeJitFunction+0×31b
0714eb8c 79e811a3 mscorwks!MethodDesc::MakeJitWorker+0×1a8
0714ebe4 79e81363 mscorwks!MethodDesc::DoPrestub+0×41b
0714ec34 01c01efe mscorwks!PreStubWorker+0xf3
WARNING: Frame IP not in any known module. Following frames may be wrong.
0714ec4c 06b08f29 0×1c01efe
0714ecb4 06b088dc 0×6b08f29
0714edf0 79e71b4c 0×6b088dc
0714edf4 79e7e45d mscorwks!CallDescrWorker+0×33
0714eeac 79e968b0 mscorwks!MethodDesc::IsSharedByGenericInstantiations+0×1c
0714ef2c 79e9eeb2 mscorwks!MetaSig::MetaSig+0×3a
0714f258 00000000 mscorwks!JIT_MonReliableEnter+0×120

If we switch to it we get a managed stack:

0:000> ~16s
eax=0714ce90 ebx=048bb528 ecx=77e63d5b edx=7ffd4000 esi=0000073c edi=00000000
eip=7c82860c esp=0714d028 ebp=0714d098 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297
ntdll!KiFastSystemCallRet:
7c82860c c3              ret

0:016> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x28c (16)
Current frame:
ChildEBP RetAddr  Caller,Callee
[...]
0714f264 792e0c3a (MethodDesc 0x79104344 +0xa System.Reflection.RuntimeMethodInfo.GetParametersNoCopy())
0714f28c 792d52d8 (MethodDesc 0x790c5058 +0x48 System.RuntimeMethodHandle.InvokeMethodFast(System.Object, System.Object[], System.Signature, System.Reflection.MethodAttributes, System.RuntimeTypeHandle))
0714f2dc 792d5086 (MethodDesc 0x791043a4 +0x106 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean))
0714f314 792d4f6e (MethodDesc 0x7910439c +0x1e System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo))
0714f338 7928ea4b (MethodDesc 0x79108798 +0x82b System.RuntimeType.InvokeMember(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object, System.Object[], System.Reflection.ParameterModifier[], System.Globalization.CultureInfo, System.String[]))
0714f478 7973ea9d (MethodDesc 0x79108264 +0x1d System.Type.InvokeMember(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object, System.Object[]))
[...]
0714f588 792d6cf6 (MethodDesc 0x791939dc +0x66 System.Threading.ThreadHelper.ThreadStart_Context(System.Object))
0714f594 792e019f (MethodDesc 0x7910276c +0x6f System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
0714f5a8 792d6c74 (MethodDesc 0x790fbde4 +0x44 System.Threading.ThreadHelper.ThreadStart())

- Dmitry Vostokov @ DumpAnalysis.org -

2 Responses to “10 Common Mistakes in Memory Analysis (Part 6)”

  1. Dmitry Vostokov Says:

    Another example, is when we are in x86 mode initially in a complete memory dump:

    16.0: kd:x86> .load wow64exts
    16.0: kd:x86> !sw
    The context is partially valid. Only x86 user-mode context is available.
    Switched to 64bit mode
    16.0: kd>

  2. Dmitry Vostokov Says:

    Some WinDbg debugger commands are context sensitive. For example, .trap command is incompatible with !running -t command: the latter shows all CPU threads having the same stack trace as in the output of k after .trap.

Leave a Reply