Archive for November 24th, 2009

Crash Dump Analysis Patterns (Part 93)

Tuesday, November 24th, 2009

Analysis of .NET managed code requires processor architectural platform specific SOS extension. For example, x64 WinDbg is not able to analyze the managed stack for a managed code exception in 32-bit process:

0:010> !analyze -v

[...]

FAULTING_IP:
kernel32!RaiseException+53
77e4bee7 5e              pop     esi

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 77e4bee7 (kernel32!RaiseException+0x00000053)
   ExceptionCode: e0434f4d (CLR exception)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 80131509

[...]

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

MANAGED_BITNESS_MISMATCH:
Managed code needs matching platform of sos.dll for proper analysis. Use ‘x86′ debugger.

[...]

0:010> kL 100
ChildEBP RetAddr 
0573f0a4 79f071ac kernel32!RaiseException+0x53
0573f104 79f0a780 mscorwks!RaiseTheExceptionInternalOnly+0x2a8
0573f1a8 058ed3b3 mscorwks!JIT_Rethrow+0xbf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0573f33c 793b0d1f <Unloaded_DllA.dll>+0x58ed3b2
0573f344 79373ecd mscorlib_ni+0x2f0d1f
0573f358 793b0c68 mscorlib_ni+0x2b3ecd
0573f370 79e7c74b mscorlib_ni+0x2f0c68
0573f380 79e7c6cc mscorwks!CallDescrWorker+0x33
0573f400 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
0573f53c 79e7c783 mscorwks!MethodDesc::CallDescr+0x19c
0573f558 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0x1f
0573f56c 79fc58cd mscorwks!MethodDescCallSite::Call_RetArgSlot+0x18
0573f754 79ef3207 mscorwks!ThreadNative::KickOffThread_Worker+0x190
0573f768 79ef31a3 mscorwks!Thread::DoADCallBack+0x32a
0573f7fc 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0573f838 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0573f860 79fc57b1 mscorwks!Thread::ShouldChangeAbortToUnload+0x33e
0573f878 79fc56ac mscorwks!ManagedThreadBase::KickOff+0x13
0573f914 79f95a2e mscorwks!ThreadNative::KickOffThread+0x269
0573ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
0573ffec 00000000 kernel32!BaseThreadStart+0x34

So we dutifully run x86 WinDbg and get the better picture of nested exceptions:

0:010> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xc68 (15)
Current frame:
ChildEBP RetAddr  Caller,Callee

EXCEPTION_OBJECT: !pe 16584f0
Exception object: 016584f0
Exception type: System.InvalidOperationException
Message: There is an error in XML document (12, 12182).

InnerException: System.IO.IOException, use !PrintException 0164f6dc to see more
[…]

StackTraceString: <none>
HResult: 80131509
There are nested exceptions on this thread. Run with -nested for details

EXCEPTION_OBJECT: !pe 164f6dc
Exception object: 0164f6dc
Exception type: System.IO.IOException
Message: Unable to read data from the transport connection: The connection was closed.

InnerException: <none>
[…]

StackTraceString: <none>
HResult: 80131620
There are nested exceptions on this thread. Run with -nested for details

MANAGED_OBJECT: !dumpobj 1655a38
Name: System.String
MethodTable: 790fd8c4
EEClass: 790fd824
Size: 270(0x10e) bytes
 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__[...]\mscorlib.dll)
String: Unable to read data from the transport connection: The connection was closed.

[...]

EXCEPTION_MESSAGE:  Unable to read data from the transport connection: The connection was closed.

MANAGED_OBJECT_NAME:  System.IO.IOException

[...]

There are other pattern instances of this kind when we need a Platform-Specific Debugger, for example, to do live debugging of an x86 process on x64 machine (needed x64 debugger) or we need to load an old 32-bit DLL extension (needed x86 debugger) for a postmortem analysis.

- Dmitry Vostokov @ DumpAnalysis.org -

Named Process: Vostokov.exe

Tuesday, November 24th, 2009

Finally you can run my moniker process (just born version doesn’t consume CPU time) and if I come across the dump of your system I would be very pleased to see Vostokov.exe in the list of running processes (!vm or !process 0 0 WinDbg commands).

lkd> !vm
[...]
0780 svchost.exe        354 (      1416 Kb)
0720 svchost.exe        330 (      1320 Kb)
0768 svchost.exe        322 (      1288 Kb)
07d4 svchost.exe        296 (      1184 Kb)
0dc8 Vostokov.exe       134 (       536 Kb)
019c smss.exe           128 (       512 Kb)
0ec4 wmplayer.exe         0 (         0 Kb)
0288 wmplayer.exe         0 (         0 Kb)
01ac wmplayer.exe         0 (         0 Kb)

lkd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffa8003bf1040
    SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 00124000  ObjectTable: fffff88000000080  HandleCount: 570.
    Image: System

[...]

PROCESS fffffa8005eeac10
    SessionId: 2  Cid: 0888    Peb: 7fffffd5000  ParentCid: 0458
    DirBase: 1c64e000  ObjectTable: fffff8800cab5b50  HandleCount: 312.
    Image: windbg.exe

PROCESS fffffa8005e87620
    SessionId: 2  Cid: 09d4    Peb: 7efdf000  ParentCid: 0f64
    DirBase: 112938000  ObjectTable: fffff8800c8b2980  HandleCount:  28.
    Image: cmd.exe

PROCESS fffffa800579cb50
    SessionId: 2  Cid: 0dc8    Peb: 7efdf000  ParentCid: 09d4
    DirBase: 092aa000  ObjectTable: fffff880105df610  HandleCount:   9.
    Image: Vostokov.exe

PROCESS fffffa8005e3e7a0
    SessionId: 2  Cid: 09c8    Peb: 7efdf000  ParentCid: 0b24
    DirBase: 78baf000  ObjectTable: fffff8800cfe0a30  HandleCount: 433.
    Image: iexplore.exe

PROCESS fffffa8005f53040
    SessionId: 2  Cid: 0db8    Peb: 7fffffd9000  ParentCid: 0458
    DirBase: 11856e000  ObjectTable: fffff8800c460710  HandleCount:  45.
    Image: notepad.exe

lkd> .process /r /p fffffa800579cb50
Implicit process is now fffffa80`0579cb50

lkd> lmv m Vostokov
start             end                 module name
00000000`001f0000 00000000`001fe000   Vostokov   (deferred)            
    Image path: c:\Users\[...]\Vostokov.exe
    Image name: Vostokov.exe
    Timestamp:        Tue Nov 24 11:19:31 2009 (4B0BC143)
    CheckSum:         000156E1
    ImageSize:        0000E000
    File version:     1.0.0.0
    Product version:  1.0.0.0
    File flags:       0 (Mask 17)
    File OS:          4 Unknown Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     1809.04b0
    CompanyName:      DumpAnalysis.org
    ProductName:      Vostokov Application
    InternalName:     Vostokov
    OriginalFilename: Vostokov.exe
    ProductVersion:   Just born
    FileVersion:      Just born
    FileDescription:  Just born Vostokov Application
    LegalCopyright:   Copyright (C) 2009 Dmitry Vostokov
    Comments:         Written by Dmitry Vostokov

You can inspect its memory if you attach WinDbg to a running instance or from a complete memory or a user process dump (symbols are supplied):

0:001> da /c 90 Vostokov!szCopyright
00000000`001fac40 "Vostokov.exe, Just born version 1.0.0.0, Copyright (c) 2009 by Dmitry Vostokov, DumpAnalysis.org"

You can download my moniker together with .cpp and .pdb files from here (named in a classic 8.3 format):

Download Vostokov.zip

Now I’m going to teach it something useful and release the next aged version soon.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 92)

Tuesday, November 24th, 2009

Sometimes the functionality of a system depends upon a specific application or service process. For example, in a database server environment it might be a database process, in printing environment it is a print spooler process or in a terminal services environment it is a terminal services process (termsvc, hosted by svchost.exe). In system failure scenarios we should check these processes for their presence (and also the presence of any coupled processes), hence the name of this pattern: Missing Process. However, if the vital process is present we should check if it is exited but references to it exist or there are any missing threads or components inside it, any suspended threads and special processes like a postmortem debugger. We shouldn’t also forget about service dependencies and their relevant process startup order. For example, we know that our service is hosted by svchost.exe and we see one such process exited but its object still referenced somewhere:

0: kd> !vm

*** Virtual Memory Usage ***
[...]
         0ed8 svchost.exe          0 (         0 Kb)
[…]

However, another command shows that it could be a different service hosted by the same image, svchost.exe, if we know that ServiceA depends on our service:

0: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8b581818  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: bff4d020  ObjectTable: e1001e18  HandleCount: 1601.
    Image: System

PROCESS 8b06d778  SessionId: none  Cid: 01a8    Peb: 7ffde000  ParentCid: 0004
    DirBase: bff4d040  ObjectTable: e13eae40  HandleCount:  22.
    Image: smss.exe

[...]

PROCESS 8aabed88  SessionId: 0  Cid: 0854    Peb: 7ffd6000  ParentCid: 0220
    DirBase: bff4d4a0  ObjectTable: e1c867a8  HandleCount: 778.
    Image: ServiceA.exe

[...]

PROCESS 8aaa6510  SessionId: 0  Cid: 0ed8    Peb: 7ffd4000  ParentCid: 0220
    DirBase: bff4d580  ObjectTable: 00000000  HandleCount:   0.
    Image: svchost.exe

[...]

Another alternative is that our service was restarted but then exited. If our process is not visible it could be possible that it was either stopped or simply crashed before.

- Dmitry Vostokov @ DumpAnalysis.org -