Archive for October 13th, 2011

Crash Dump Analysis Patterns (Part 151)

Thursday, October 13th, 2011

When disassembling JIT code it is good to see annotated function calls with full type and token information:

0:000> !CLRStack
OS Thread Id: 0xbf8 (0)
ESP       EIP
001fef90 003200a4 ClassMain.DoWork()
001fef94 00320082 ClassMain.Main(System.String[])
001ff1b0 79e7c74b [GCFrame: 001ff1b0]

0:000> !U 00320082
Normal JIT generated code
ClassMain.Main(System.String[])
Begin 00320070, size 13
00320070 b960300d00      mov     ecx,0D3060h (MT: ClassMain)
00320075 e8a21fdaff      call    000c201c (JitHelp: CORINFO_HELP_NEWSFAST)
0032007a 8bc8            mov     ecx,eax
0032007c ff159c300d00    call    dword ptr ds:[0D309Ch] (ClassMain.DoWork(), mdToken: 06000002)
>>> 00320082 c3              ret

However, this doesn’t work when we disable the output of raw bytes:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> !U 00320082
Normal JIT generated code
ClassMain.Main(System.String[])
Begin 00320070, size 13
00320070 mov     ecx,0D3060h
00320075 call    000c201c
0032007a mov     ecx,eax
0032007c call    dword ptr ds:[0D309Ch]
>>> 00320082 ret

Here we can still double check JIT-ed function calls manually:

0:000> dd 0D309Ch l1
000d309c  00320098

0:000> !IP2MD 00320098
MethodDesc: 000d3048
Method Name: ClassMain.DoWork()
Class: 000d1180
MethodTable: 000d3060
mdToken: 06000002
Module: 000d2c3c
IsJitted: yes
m_CodeOrIL: 00320098

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Assembly Language, Crash Dump Analysis, Crash Dump Patterns, Debugging, Reverse Engineering | No Comments »

Crash Dump Analysis (Part 42j)

Thursday, October 13th, 2011

This is an additional variation of the general Wait Chain pattern where mutexes (mutants) are involved in thread wait chains, for example:

THREAD fffffa8019388b60  Cid 02e8.cfd0  Teb: 000007fffffa2000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa800d75daf0  Mutant - owning thread fffffa800ea2ab60
[...]

THREAD fffffa8016abab60  Cid 02e8.ec34  Teb: 000007fffffae000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa800d75daf0  Mutant - owning thread fffffa800ea2ab60
[...]

We have seen such dependencies in various previous pattern interaction case studies such as:

- Inconsistent dump, stack trace collection, LPC, thread, process, executive resource wait chains, missing threads and waiting thread time

- Inconsistent dump, blocked threads, wait chains, incorrect stack trace and process factory

- Semantic Split pattern example

- Insufficient memory, handle leak, wait chain, deadlock, inconsistent dump and overaged system

- Blocked GUI thread, wait chain and virtualized process

- LPC/ALPC Wait Chain pattern example

- Mixed object Deadlock pattern example

- LPC Deadlock pattern example

Another example I show here is an unusual number of mutant dependencies in one complete memory dump from hang system:

AppA(KTHREAD-1)  -> AppB(KTHREAD-2)
AppB(KTHREAD-3)  -> ServiceA(KTHREAD-4)     
AppA(KTHREAD-5)  -> ServiceA(KTHREAD-4)
AppB(KTHREAD-6)  -> AppA(KTHREAD-7)
AppB(KTHREAD-6)  -> AppC(KTHREAD-8)
AppC(KTHREAD-9)  -> ServiceA(KTHREAD-4)
AppC(KTHREAD-10) -> AppB(KTHREAD-11)

Here the notation AppX(N)->AppY(M) means that a thread N from AppX process is waiting for a mutant that is owned by a thread M from AppY process. Because AppB, AppC and ServiceA belonged to the Same Vendor it was advised to check with that ISV.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »