Crash Dump Analysis Patterns (Part 2c)
Monday, October 24th, 2011Here’s a managed heap counterpart to process heap and kernel pool corruption patterns. It is usually detected by CLR during GC phases. Here is a typical stack from CLR 2 (CLR 4 is similar):
0:000> kL
ChildEBP RetAddr
002baae0 779b06a0 ntdll!KiFastSystemCallRet
002baae4 772c77d4 ntdll!NtWaitForSingleObject+0xc
002bab54 772c7742 kernel32!WaitForSingleObjectEx+0xbe
002bab68 7a0c0a43 kernel32!WaitForSingleObject+0x12
002bab98 7a0c0e89 mscorwks!ClrWaitForSingleObject+0x24
002bb054 7a0c2bfd mscorwks!RunWatson+0x1df
002bb798 7a0c3171 mscorwks!DoFaultReportWorker+0xb62
002bb7d4 7a106b2d mscorwks!DoFaultReport+0xc3
002bb7fc 7a1061ac mscorwks!WatsonLastChance+0x43
002bbcb8 7a10624f mscorwks!EEPolicy::LogFatalError+0x3ae
002bbcd0 79ffee2f mscorwks!EEPolicy::HandleFatalError+0x36
002bbcf4 79f04f1f mscorwks!CLRVectoredExceptionHandlerPhase3+0xc1
002bbd28 79f04e98 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
002bbd5c 79f9149e mscorwks!CLRVectoredExceptionHandler+0x10a
002bbd70 779b1039 mscorwks!CLRVectoredExceptionHandlerShimX86+0x27
002bbd94 779b100b ntdll!ExecuteHandler2+0x26
002bbe3c 779b0e97 ntdll!ExecuteHandler+0x24
002bbe3c 79f69360 ntdll!KiUserExceptionDispatcher+0xf
002bc13c 79f663f1 mscorwks!SVR::heap_segment_next_rw+0xf
002bc228 79f65d63 mscorwks!WKS::gc_heap::plan_phase+0×37c
002bc248 79f6614c mscorwks!WKS::gc_heap::gc1+0×6e
002bc25c 79f65f5d mscorwks!WKS::gc_heap::garbage_collect+0×261
002bc288 79f663c2 mscorwks!WKS::GCHeap::GarbageCollectGeneration+0×1a9
002bc314 79ef1566 mscorwks!WKS::gc_heap::try_allocate_more_space+0×12e
002bc328 79ef1801 mscorwks!WKS::gc_heap::allocate_more_space+0×11
002bc348 79e7510e mscorwks!WKS::GCHeap::Alloc+0×3b
002bc364 79e86713 mscorwks!Alloc+0×60
002bc3a0 79e86753 mscorwks!SlowAllocateString+0×29
002bc3ac 79eb4efb mscorwks!UnframedAllocateString+0xc
002bc3e0 79e91f58 mscorwks!AllocateStringObject+0×2e
002bc424 79e82892 mscorwks!GlobalStringLiteralMap::AddStringLiteral+0×3f
002bc438 79e82810 mscorwks!GlobalStringLiteralMap::GetStringLiteral+0×43
002bc47c 79e82956 mscorwks!AppDomainStringLiteralMap::GetStringLiteral+0×72
002bc494 79e81b6f mscorwks!BaseDomain::GetStringObjRefPtrFromUnicodeString+0×31
002bc4cc 79ef4704 mscorwks!Module::ResolveStringRef+0×88
002bc4e4 79f23132 mscorwks!ConstructStringLiteral+0×39
002bc558 7908c351 mscorwks!CEEInfo::constructStringLiteral+0×108
002bc57c 7906276d mscorjit!Compiler::fgMorphConst+0xa3
002bc598 79065ea0 mscorjit!Compiler::fgMorphTree+0×63
002bc610 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc63c 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc658 79065ea0 mscorjit!Compiler::fgMorphTree+0xa3
002bc6d0 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc6fc 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc718 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bc738 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bc774 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bc788 79064e63 mscorjit!Compiler::fgMorph+0×60
002bc798 790614e6 mscorjit!Compiler::compCompile+0×5f
002bc7f0 79061236 mscorjit!Compiler::compCompile+0×2df
002bc884 7906118c mscorjit!jitNativeCode+0xb8
002bc8bc 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002bc928 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002bc96c 79f0f8da mscorwks!invokeCompileMethod+0×31
002bc9c4 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002bcd7c 79f0e795 mscorwks!UnsafeJitFunction+0×230
002bce20 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002bce78 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002bcec8 00330836 mscorwks!PreStubWorker+0xeb
WARNING: Frame IP not in any known module. Following frames may be wrong.
002bcee0 79e7c74b 0×330836
002bcf10 79e7c6cc mscorwks!CallDescrWorker+0×33
002bcf90 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
002bd0d0 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
002bd0ec 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
002bd100 79e8b983 mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
002bd1d8 79e8b8e6 mscorwks!MethodTable::RunClassInitWorker+0×8b
002bd260 79e8b7fa mscorwks!MethodTable::RunClassInitEx+0×11e
002bd724 79ebcee6 mscorwks!MethodTable::DoRunClassInitThrowing+0×2f0
002bd79c 79fc49db mscorwks!MethodTable::CheckRunClassInitNT+0×8c
002bd82c 790a2801 mscorwks!CEEInfo::initClass+0×19b
002bddcc 79062cdc mscorjit!Compiler::impExpandInline+0×2aaa
002bde24 79062b7c mscorjit!Compiler::fgMorphCallInline+0xf8
002bde50 7906311f mscorjit!Compiler::fgMorphCall+0×27b
002bde6c 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bde8c 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bdec8 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bdedc 79064e63 mscorjit!Compiler::fgMorph+0×60
002bdeec 790614e6 mscorjit!Compiler::compCompile+0×5f
002bdf44 79061236 mscorjit!Compiler::compCompile+0×2df
002bdfd8 7906118c mscorjit!jitNativeCode+0xb8
002be010 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002be07c 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002be0c0 79f0f8da mscorwks!invokeCompileMethod+0×31
002be118 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002be4d0 79f0e795 mscorwks!UnsafeJitFunction+0×230
002be574 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002be5cc 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002be61c 00330836 mscorwks!PreStubWorker+0xeb
002be634 0570c859 0×330836
002be69c 0595bcc1 0×570c859
002be700 0595b954 0×595bcc1
002be704 099b66e0 0×595b954
002be708 002be728 0×99b66e0
002be70c 09589c90 0×2be728
002be728 099b67b8 0×9589c90
002be72c 00000000 0×99b67b8
Usually !VerifyHeap SOS command helps to find the first invalid object on managed heap and shows the last valid one. Sometimes the corruption can deeply affect heap or when a crash happens during traversal GC state might not be valid for analysis:
0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state
0:000> !DumpHeap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state
In such cases it is recommended to collect several dumps to catch more consistent heap state:
0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
object 0981f024: does not have valid MT
curr_object: 0981f024
Last good object: 0981f010
—————-
Then we can use !DumpObj (!do) command to check objects and d* WinDbg commands to inspect raw memory.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -