Archive for August, 2018

Crash Dump Analysis Patterns (Part 8c)

Saturday, August 11th, 2018

For completion, we introduce a managed space version of Hidden Exception in addition to user and kernel space variants.

0:000> ~*kL
[...]
13 Id: 1b70.1c2c Suspend: 0 Teb: 00446000 Unfrozen
# ChildEBP RetAddr
00 08e7ec4c 755e1cf3 ntdll!NtWaitForMultipleObjects+0xc
01 08e7ede0 6ef8bc6e KERNELBASE!WaitForMultipleObjectsEx+0x133
02 08e7ee30 6ef8b9b3 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x3c
03 08e7eebc 6ef8baa4 clr!Thread::DoAppropriateWaitWorker+0x237
04 08e7ef28 6ef8bc14 clr!Thread::DoAppropriateWait+0x64
05 08e7ef74 6eef648b clr!CLREventBase::WaitEx+0x128
06 08e7ef8c 6f0058f6 clr!CLREventBase::Wait+0x1a
07 08e7f018 6f005834 clr!AwareLock::EnterEpilogHelper+0xa8
08 08e7f060 6f005980 clr!AwareLock::EnterEpilog+0x48
09 08e7f078 6f00662c clr!AwareLock::Enter+0x4a
0a 08e7f104 08d71d79 clr!JITutil_MonEnterWorker+0x9c
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b 08e7f120 6dd9608d 0x8d71d79
0c 08e7f12c 6ddc2925 mscorlib_ni+0x3c608d
0d 08e7f190 6ddc2836 mscorlib_ni+0x3f2925
0e 08e7f1a4 6ddc27f1 mscorlib_ni+0x3f2836
0f 08e7f1c0 6dd95fe8 mscorlib_ni+0x3f27f1
10 08e7f1d8 6ee6eaf6 mscorlib_ni+0x3c5fe8
11 08e7f1e4 6ee71d50 clr!CallDescrWorkerInternal+0x34
12 08e7f238 6ee77764 clr!CallDescrWorkerWithHandler+0x6b
13 08e7f2a0 6eef4d2d clr!MethodDescCallSite::CallTargetWorker+0x16a
14 08e7f414 6efae269 clr!ThreadNative::KickOffThread_Worker+0x173
15 08e7f428 6efae2d3 clr!ManagedThreadBase_DispatchInner+0x71
16 08e7f4cc 6efae3a0 clr!ManagedThreadBase_DispatchMiddle+0x7e
17 08e7f528 6ee7af05 clr!ManagedThreadBase_DispatchOuter+0x5b
18 08e7f534 6ee7aea2 clr!ManagedThreadBase_DispatchInCorrectAD+0x15
19 08e7f600 6ee7af4d clr!Thread::DoADCallBack+0x328
1a 08e7f624 6efae2d3 clr!ManagedThreadBase_DispatchInner+0x4e
1b 08e7f6c8 6efae3a0 clr!ManagedThreadBase_DispatchMiddle+0x7e
1c 08e7f724 6efae40f clr!ManagedThreadBase_DispatchOuter+0x5b
1d 08e7f748 6eef4be2 clr!ManagedThreadBase_FullTransitionWithAD+0x2f
1e 08e7f7c4 6eef62d1 clr!ThreadNative::KickOffThread+0x256
1f 08e7fbe4 76c28484 clr!Thread::intermediateThreadProc+0x55
20 08e7fbf8 77842fea kernel32!BaseThreadInitThunk+0x24
21 08e7fc40 77842fba ntdll!__RtlUserThreadStart+0x2f
22 08e7fc50 00000000 ntdll!_RtlUserThreadStart+0x1b
[...]

0:000> ~13s
eax=00000000 ebx=00000001 ecx=00000000 edx=00000000 esi=00000001 edi=00000001
eip=7784a7bc esp=08e7ec50 ebp=08e7ede0 iopl=0 nv up ei pl nz ac pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216
ntdll!NtWaitForMultipleObjects+0xc:
7784a7bc c21400 ret 14h

0:013> !CLRStack
OS Thread Id: 0x1c2c (13)
Child SP IP Call Site
08e7efb4 7784a7bc [GCFrame: 08e7efb4]
08e7f094 7784a7bc [HelperMethodFrame_1OBJ: 08e7f094] System.Threading.Monitor.Enter(System.Object)
08e7f10c 08d71d79 UserQuery+ClassMain.thread_proc_1()
08e7f128 6dd9608d *** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
08e7f134 6ddc2925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
08e7f1a0 6ddc2836 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
08e7f1b4 6ddc27f1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
08e7f1cc 6dd95fe8 System.Threading.ThreadHelper.ThreadStart()
08e7f308 6ee6eaf6 [GCFrame: 08e7f308]
08e7f4e8 6ee6eaf6 [DebuggerU2MCatchHandlerFrame: 08e7f4e8]
08e7f554 6ee6eaf6 [ContextTransitionFrame: 08e7f554]
08e7f6e4 6ee6eaf6 [DebuggerU2MCatchHandlerFrame: 08e7f6e4]

0:013> !teb
TEB at 00446000
ExceptionList: 08e7edd0
StackBase: 08e80000
StackLimit: 08e7a000

SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 00446000
EnvironmentPointer: 00000000
ClientId: 00001b70 . 00001c2c
RpcHandle: 00000000
Tls Storage: 008eb8e8
PEB Address: 0040a000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0

0:013> !DumpStackObjects 08e7a000 08e80000
OS Thread Id: 0×1c2c (13)
ESP/REG Object Name
08E7DD18 0270f714 LINQPad.ExecutionModel.OutPipe
08E7DD20 02736ca8 LINQPad.Disposable
08E7DD2C 0270f714 LINQPad.ExecutionModel.OutPipe
08E7DD3C 02736ca8 LINQPad.Disposable
08E7DD40 02736c88 System.Action
08E7DD44 02736ca8 LINQPad.Disposable
08E7DD64 0270f714 LINQPad.ExecutionModel.OutPipe
08E7DD98 02736ca8 LINQPad.Disposable
08E7DDB8 0270f714 LINQPad.ExecutionModel.OutPipe
08E7DE78 0270f9ec System.Object
08E7DE7C 0270f990 LINQPad.ObjectGraph.Formatters.HtmlWriter
08E7DEAC 0270f990 LINQPad.ObjectGraph.Formatters.HtmlWriter
08E7DEE4 0262e16c System.String
08E7DEF8 026aa9d0 System.String
08E7DF04 0270f990 LINQPad.ObjectGraph.Formatters.HtmlWriter
08E7E054 02724ecc System.Threading.ThreadHelper
08E7E058 026fad7c System.Threading.ContextCallback
08E7E06C 026fad7c System.Threading.ContextCallback
08E7E074 02724ecc System.Threading.ThreadHelper
08E7E0A8 0272fb68 System.NullReferenceException
08E7E0AC 026fad7c System.Threading.ContextCallback
08E7E0B8 02724ecc System.Threading.ThreadHelper
08E7E340 0272fcc0 System.Runtime.CompilerServices.RuntimeHelpers+TryCode
08E7E344 0272fce0 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
08E7E348 0272fca4 System.Environment+ResourceHelper+GetResourceStringUserData
08E7E35C 0272fce0 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
08E7E378 0272fca4 System.Environment+ResourceHelper+GetResourceStringUserData
08E7E37C 0272fc0c System.Environment+ResourceHelper
08E7E964 0272fb68 System.NullReferenceException
08E7EB3C 02724ecc System.Threading.ThreadHelper
08E7ECCC 02724ecc System.Threading.ThreadHelper
08E7ECD0 026fad7c System.Threading.ContextCallback
08E7ECD8 0272fa88 System.String critical section 1
08E7EFE8 0272fabc System.String critical section 2
08E7F034 026fad7c System.Threading.ContextCallback
08E7F088 02724ecc System.Threading.ThreadHelper
08E7F08C 026fad7c System.Threading.ContextCallback
08E7F0B8 02724ecc System.Threading.ThreadHelper
08E7F0C0 026fad7c System.Threading.ContextCallback
08E7F0F0 0272fabc System.String critical section 2
08E7F11C 026fad7c System.Threading.ContextCallback
08E7F128 02724f00 System.Threading.ExecutionContext
08E7F134 02724e98 System.Threading.Thread
08E7F144 02724e98 System.Threading.Thread
08E7F188 02724f00 System.Threading.ExecutionContext
08E7F18C 026fad7c System.Threading.ContextCallback
08E7F19C 02724ecc System.Threading.ThreadHelper
08E7F1B0 02724ecc System.Threading.ThreadHelper
08E7F1B8 02724ecc System.Threading.ThreadHelper
08E7F1BC 02724f00 System.Threading.ExecutionContext
08E7F1C8 02724ecc System.Threading.ThreadHelper
08E7F244 02724ee0 System.Threading.ThreadStart
08E7F2C4 02724ee0 System.Threading.ThreadStart
08E7F2D8 02724ee0 System.Threading.ThreadStart

The example dump can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 256)

Saturday, August 4th, 2018

Backwards disassembling used in memory analysis patterns such as Coincidental Symbolic Information may be ambiguous and can show Wild Code output. This may also be debugger disassembling algorithm dependent. For example, default 8-instruction backwards disassembly shows this code:

0:011> ub 00007ff8`cdc9b4bf
00007ff8`cdc9b4ab 855948 test dword ptr [rcx+48h],ebx
00007ff8`cdc9b4ae b988bf03a8 mov ecx,0A803BF88h
00007ff8`cdc9b4b3 f4 hlt
00007ff8`cdc9b4b4 0100 add dword ptr [rax],eax
00007ff8`cdc9b4b6 00488b add byte ptr [rax-75h],cl
00007ff8`cdc9b4b9 09e8 or eax,ebp
00007ff8`cdc9b4bb 117236 adc dword ptr [rdx+36h],esi
00007ff8`cdc9b4be 5f pop rdi

However, if we specify the number of instructions to disassemble except 7 and 8 we get a different result (which is more correct from the forward code execution view since we disassembled the saved return address from the stack region):

0:011> ub 00007ff8`cdc9b4bf L1
00007ff8`cdc9b4ba e81172365f call clr!JIT_MonEnter (00007ff9`2d0026d0)

0:011> ub 00007ff8`cdc9b4bf L2
00007ff8`cdc9b4b7 488b09 mov rcx,qword ptr [rcx]
00007ff8`cdc9b4ba e81172365f call clr!JIT_MonEnter (00007ff9`2d0026d0)

0:011> k L10
# Child-SP RetAddr Call Site
00 0000002a`fc23e308 00007ff9`53d06099 ntdll!NtWaitForMultipleObjects+0x14
01 0000002a`fc23e310 00007ff9`2d1a96be KERNELBASE!WaitForMultipleObjectsEx+0xf9
02 0000002a`fc23e610 00007ff9`2d1a951c clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62
03 0000002a`fc23e670 00007ff9`2d1a9315 clr!Thread::DoAppropriateWaitWorker+0x1e4
04 0000002a`fc23e770 00007ff9`2d0c2b7f clr!Thread::DoAppropriateWait+0x7d
05 0000002a`fc23e7f0 00007ff9`2d1aa491 clr!CLREventBase::WaitEx+0xc4
06 0000002a`fc23e880 00007ff9`2d1aa39e clr!AwareLock::EnterEpilogHelper+0xc2
07 0000002a`fc23e940 00007ff9`2d1c1a92 clr!AwareLock::EnterEpilog+0x62
08 0000002a`fc23e9a0 00007ff8`cdc9b4bf clr!JITutil_MonEnterWorker+0xe2
09 0000002a`fc23eb40 00007ff9`275231d3 0×00007ff8`cdc9b4bf
0a 0000002a`fc23eb80 00007ff9`27523064 mscorlib_ni+0×5031d3
0b 0000002a`fc23ec50 00007ff9`27523032 mscorlib_ni+0×503064
0c 0000002a`fc23ec80 00007ff9`2751c812 mscorlib_ni+0×503032
0d 0000002a`fc23ecd0 00007ff9`2d006bb3 mscorlib_ni+0×4fc812
0e 0000002a`fc23ed10 00007ff9`2d006a70 clr!CallDescrWorkerInternal+0×83
0f 0000002a`fc23ed50 00007ff9`2d00735d clr!CallDescrWorkerWithHandler+0×4e

We call this analysis pattern Disassembly Ambiguity. The example dump can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -