Software Diagnostics Library

Today we introduce an icon for Manual Dump (process) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This forthcoming full color book is the complete transcript of a Webinar organized by Memory Dump Analysis Services (www.DumpAnalysis.com).

It discusses user vs. kernel vs. physical (complete) memory space, challenges of complete memory dump analysis, common WinDbg commands, patterns and pattern-driven analysis methodology, common mistakes, fiber bundles, DumpAnalysis.org case studies and illustrates step by step a hands-on exercise in a complete memory dump analysis.

• Title: Fundamentals of Complete Crash and Hang Memory Dump Analysis
• Author: Dmitry Vostokov
• Publisher: OpenTask (October 2010)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 48 pages
• ISBN-13: 978-1906717155

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Manual Dump (kernel) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally I compiled a Questions and Answers page with all necessary links and examples during the weekend:

http://www.dumpanalysis.com/FCMDA-Q-A

I also added text versions of logs (in addition to zip files) to a Webinar materials page:

http://www.dumpanalysis.com/FCMDA-materials

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Special Stack Trace pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Like in a literature (and in a narratology in general) we have components that trace themselves and components that tell the story of a computation including status updates they query about other components and subsystems. This pattern gets its name Diegetic Messages from diegesis. Here’s the difference between diegetic (in blue bold) and non-diegetic trace messages:

PID    TID    TIME          MESSAGE     
11864  11912  06:34:53.598  ModuleA: foo called bar. Status OK.
11620  10372  06:34:59.754  ModuleB: ModuleA integrity check. Status OK.

Some modules may emit messages that tell about their status but from their message text we know the larger computation story like in a process startup sequence example.  

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The online version of Debugged! MZ/PE) underwent the total redesign:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Custom Exception Handler (user space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having encountered several crash dumps with the code running on heap and the following similar stack traces

1: kd> k
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
02cdfbfc 0056511a 0x634648
02cdfc24 005651a1 ModuleA!ClassA::~ClassA+0x5a
02cdfc30 00562563 ModuleA!ClassA::`scalar deleting destructor'+0x11
[...]
02cdffec 00000000 kernel32!BaseThreadStart+0x37

I decided to model this situation. The idea was to corrupt a class member by overriding its vtable pointer with a heap entry address. Because the virtual destructor address is a first virtual method table entry in our class memory layout I made sure that it points to the same heap address by making vtable pointer a dereference fixpoint. Here is a source code based on how Visual C++ compiler implements objects in memory:

class Member {
public:
    virtual ~Member() { data = 1; };
public:
    int data;
};

class Compound {
public:
    Compound(): pm(NULL) { pm = new Member(); }
    virtual ~Compound() { delete pm; }
    void Corrupt() {
        unsigned int * pbuf = new unsigned int[0x10];
        *pbuf = reinterpret_cast<unsigned int>(pbuf); // to ensure that the code would run through pbuf pointer
        *reinterpret_cast<unsigned int *>(pm) = reinterpret_cast<unsigned int>(pbuf);
    }
    Member *pm;
};

int _tmain(int argc, _TCHAR* argv[])
{
    Compound *pc = new Compound();
    pc->Corrupt();
    delete pc;
    return 0;
}

In a crash dump we therefore see the similar stack trace:

0:000> .ecxr
eax=001f4c28 ebx=7efde000 ecx=001f4c18 edx=001f4c28 esi=00000000 edi=00000000
eip=001f4c28 esp=003cf7d0 ebp=003cf7e8 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
001f4c28 284c1f00        sub     byte ptr [edi+ebx],cl      ds:002b:7efde000=00

0:000> k
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
003cf7cc 011d10e5 0×1f4c28
003cf7e8 011d114f Destructors!Compound::~Compound+0×35
003cf7f4 011d121e Destructors!Compound::`scalar deleting destructor’+0xf
003cf82c 011d1498 Destructors!wmain+0×8e
003cf874 77043677 Destructors!__tmainCRTStartup+0xfa
003cf880 77719d72 kernel32!BaseThreadInitThunk+0xe
003cf8c0 77719d45 ntdll!__RtlUserThreadStart+0×70
003cf8d8 00000000 ntdll!_RtlUserThreadStart+0×1b

We now check the correctness of the stack trace by examining the return addresses:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> ub 011d10e5
Destructors!Compound::~Compound+0×21:
011d10d1 cmp  dword ptr [ebp-4],0
011d10d5 je   Destructors!Compound::~Compound+0×3a (011d10ea)
011d10d7 push 1
011d10d9 mov  ecx,dword ptr [ebp-4]
011d10dc mov  edx,dword ptr [ecx]
011d10de mov  ecx,dword ptr [ebp-4]
011d10e1 mov  eax,dword ptr [edx]
011d10e3 call eax

0:000> ub 011d114f
Destructors!Compound::Corrupt+0×3e:
011d113e int  3
011d113f int  3
Destructors!Compound::`scalar deleting destructor’:
011d1140 push ebp
011d1141 mov  ebp,esp
011d1143 push ecx
011d1144 mov  dword ptr [ebp-4],ecx
011d1147 mov  ecx,dword ptr [ebp-4]
011d114a call Destructors!Compound::~Compound (011d10b0)

We now examine the crash address:

0:000> u 001f4c28
001f4c28 sub  byte ptr [edi+ebx],cl
001f4c2c les  eax,fword ptr [eax]
001f4c2e pop  ds
001f4c2f add  byte ptr [eax],al
001f4c31 add  byte ptr [eax],al
001f4c33 add  byte ptr [eax],al
001f4c35 add  byte ptr [eax],al
001f4c37 add  byte ptr [eax],al

Then we check that it resides in a heap segment:

0:000> dt _PEB 7efde000
Destructors!_PEB
[...]
  +0x088 NumberOfHeaps : 2
  +0x08c MaximumNumberOfHeaps : 0x10
  +0x090 ProcessHeaps : 0x777e4740 -> 0x004b0000 Void
[...]

0:000> dd 0x777e4740 l2
777e4740 004b0000 001f0000

0:000> !heap 001f0000
Index Address Name Debugging options enabled
  2: 001f0000
  Segment at 001f0000 to 00200000 (00005000 bytes committed)

Now we check vtable to see that it was normal for Compound object but corrupt for Member object:

0:000> .frame 1
01 003cf7e8 011d114f Destructors!Compound::~Compound+0x35

0:000> dv /i /V
prv local  003cf7dc @ebp-0x0c            this = 0x001f4c08
0:000> dt Destructors!Compound 0x001f4c08
   +0x000 __VFN_table : 0x011daa0c
   +0x004 pm               : 0x001f4c18 Member

0:000> dps 0x001f4c08 l1
001f4c08  011daa0c Destructors!Compound::`vftable'

0:000> dps 0x001f4c18 l1
001f4c18  001f4c28

0:000> dps 001f4c28 l1
001f4c28  001f4c28

The application, its source code and PDB file are available for download here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Madness, the absence of memory dumps.

Michel Foucault

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today I celebrated 4 years of blogging that resulted in more than 2,000 posts across 8 blogs. I would like to thank everyone for their continuing support!

The updated timeline

This blog post belongs to the 5th year of blogging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The whole code is a series of balanced defects. Fix one and the program crashes.

Ralph Waldo Emerson

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump is a double buzzword.

Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes a software trace is truncated when the trace session was stopped prematurely, often when a problem didn’t manifest itself visually. We can diagnose such traces by their short time duration, missing anchor messages or components necessary for analysis. My favourite example is user session initialization in a Citrix terminal services environment when problem effects are visible only after the session is fully initialized and an application is launched but a truncated CDF trace only shows the launch of winlogon.exe despite the presence of a process creation trace provider or other components that record the process launch sequence and the trace itself lasts only a few seconds after that.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Coupled Processes (semantics) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The one obvious pattern that is shown in many case studies on this blog is Exception Stack Trace (or Exception Thread). This is a stack trace that has exception processing functions, for example:

   9  Id: 1df4.a08 Suspend: -1 Teb: 7fff4000 Unfrozen
ChildEBP RetAddr 
1022f5a8 7c90df4a ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 kernel32!UnhandledExceptionFilter+0×8b9
1022f908 7c839b39 kernel32!BaseThreadStart+0×4d
1022f930 7c9032a8 kernel32!_except_handler3+0×61
1022f954 7c90327a ntdll!ExecuteHandler2+0×26
1022fa04 7c90e48a ntdll!ExecuteHandler+0×24
1022fa04 7c812afb ntdll!KiUserExceptionDispatcher+0xe
1022fd5c 0b82e680 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
1022fd94 0b82d2f2 DllA+0×21e640
1022fde8 7753004f DllA+0×21d4f2
1022fdfc 7753032f ole32!CClassCache::CDllPathEntry::CanUnload_rl+0×3b
1022ff3c 7753028b ole32!CClassCache::FreeUnused+0×70
1022ff4c 775300b5 ole32!CoFreeUnusedLibrariesEx+0×36
1022ff58 77596af5 ole32!CoFreeUnusedLibraries+0×9
1022ff6c 77566ff9 ole32!CDllHost::MTAWorkerLoop+0×25
1022ff8c 7752687c ole32!CDllHost::WorkerThread+0xc1
1022ff94 774fe3ee ole32!DLLHostThreadEntry+0xd
1022ffa8 774fe456 ole32!CRpcThread::WorkerLoop+0×1e
1022ffb4 7c80b729 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0×1b
1022ffec 00000000 kernel32!BaseThreadStart+0×37

Such exceptions can be detected by the default analysis command (for example, !analyze -v WinDbg command) or by inspecting a stack trace collection. However if we didn’t see any exception thread it doesn’t mean that no exception had occurred. There could be hidden exceptions on raw stack data. 

In our case we can get the exception information by looking at parameters to in unhandled exception filter:

0:009> kv 3
ChildEBP RetAddr Args to Child
1022f5a8 7c90df4a 7c8648a2 00000002 1022f730 ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 00000002 1022f730 00000001 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 1022f928 7c839b39 1022f930 kernel32!UnhandledExceptionFilter+0×8b9

0:009> .exptr 1022f928

----- Exception record at 1022fa1c:
ExceptionAddress: 7c812afb (kernel32!RaiseException+0x00000053)
  ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
  Parameter[0]: 19930520
  Parameter[1]: 1022fda4
  Parameter[2]: 0b985074
  pExceptionObject: 1022fda4
  _s_ThrowInfo : 0b985074

----- Context record at 1022fa3c:
eax=1022fd0c ebx=00000001 ecx=00000000 edx=1022fda4 esi=1022fd94 edi=77606068
eip=7c812afb esp=1022fd08 ebp=1022fd5c iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
kernel32!RaiseException+0x53:
7c812afb 5e pop esi

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we get memory dumps that are difficult to analyze in full because some if not most of information was omitted during saving. These are usually small memory dumps (contrasted with kernel and complete) and user process minidumps. We can easily recognize that when we open a dump:

User Mini Dump File: Only registers, stack and portions of memory are available

or

Mini Kernel Dump File: Only registers and stack trace are available

The same also applies to user dumps where thread times information is omitted so it is not possible to use !runaway WinDbg command or to a dump saved with various options of .dump command (including privacy-aware) instead of /ma or deprecated /f option. On the contrary, manually erased data in crash dumps looks more like an example of another pattern called Lateral Damage.

The similar cases of abridged dumps are discussed in Wrong Dump and Missing Space antipatterns.

Anyway, we shouldn’t dismiss such dumps and should try to analyze them. For example, some approaches (including using image binaries) are listed in kernel minidump analysis series. We can even see portions of raw stack data in search of execution residue:

0: kd> !thread
GetPointerFromAddress: unable to read from 81d315b0
THREAD 82f49020  Cid 0004.0034  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    Unable to read nt!_IRP @ 8391e008
Not impersonating
GetUlongFromAddress: unable to read from 81d0ad90
Owning Process            82f00ab0       Image:         System
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      4000214     
Context Switch Count      21886            
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!ExpWorkerThread (0x81c78ea3)
Stack Init 85be0000 Current 85bdf7c0 Base 85be0000 Limit 85bdd000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
[…]

0: kd> dps 85bdd000 85be0000
85bdd000  ????????
85bdd004  ????????
85bdd008  ????????
85bdd00c  ????????
85bdd010  ????????
85bdd014  ????????
85bdd018  ????????
85bdd01c  ????????
85bdd020  ????????
85bdd024  ????????
85bdd028  ????????
[...]
85bdf8c4  ????????
85bdf8c8  ????????
85bdf8cc  ????????
85bdf8d0  0000000a
85bdf8d4  a112883e
85bdf8d8  0000001b
85bdf8dc  00000000
85bdf8e0  81c28750 nt!KeSetEvent+0x4d
85bdf8e4  85bdf8e8
85bdf8e8  85bdf970
85bdf8ec  81c28750 nt!KeSetEvent+0x4d
85bdf8f0  badb0d00
85bdf8f4  00000000
85bdf8f8  00000000
85bdf8fc  81cf4820 nt!KiInitialPCR+0x120
85bdf900  00000000
85bdf904  85bdf938
85bdf908  81cf4820 nt!KiInitialPCR+0x120
85bdf90c  00000000
85bdf910  81d32300 nt!IopTimerLock
85bdf914  00000000
85bdf918  81fa0000 nt!_NULL_IMPORT_DESCRIPTOR <PERF> (nt+0x3a0000)
85bdf91c  85bd0023
85bdf920  00000023
85bdf924  00000000
85bdf928  81d323c0 nt!KiDispatcherLock
85bdf92c  a1128828
85bdf930  85bdf9b4
85bdf934  85bdfdb0
85bdf938  00000030
85bdf93c  84ca6f40
85bdf940  84ca6f38
85bdf944  00000001
85bdf948  85bdf970
85bdf94c  00000000
85bdf950  81c28750 nt!KeSetEvent+0x4d
85bdf954  00000008
85bdf958  00010246
85bdf95c  00000000
85bdf960  84ca68a0
[...]
85bdfd2c  82f49020
85bdfd30  835ca4d0
85bdfd34  a6684538
85bdfd38  81cfde7c nt!ExWorkerQueue+0x3c
85bdfd3c  00000001
85bdfd40  00000000
85bdfd44  85bdfd7c
85bdfd48  81c78fa0 nt!ExpWorkerThread+0xfd
85bdfd4c  835ca4d0
85bdfd50  00000000
85bdfd54  82f49020
85bdfd58  00000000
85bdfd5c  00000000
85bdfd60  0069000b
85bdfd64  00000000
85bdfd68  00000001
85bdfd6c  00000000
85bdfd70  835ca4d0
85bdfd74  81da9542 nt!PnpDeviceEventWorker
85bdfd78  00000000
85bdfd7c  85bdfdc0
85bdfd80  81e254e0 nt!PspSystemThreadStartup+0x9d
85bdfd84  835ca4d0
85bdfd88  85bd4680
85bdfd8c  00000000
85bdfd90  00000000
85bdfd94  00000000
85bdfd98  00000002
85bdfd9c  00000000
85bdfda0  00000000
85bdfda4  00000001
85bdfda8  85bdfd88
85bdfdac  85bdfdbc
85bdfdb0  ffffffff
85bdfdb4  81c8aad5 nt!_except_handler4
85bdfdb8  81c9ddb8 nt!`string'+0x4
85bdfdbc  00000000
85bdfdc0  00000000
85bdfdc4  81c9159e nt!KiThreadStartup+0x16
85bdfdc8  81c78ea3 nt!ExpWorkerThread
85bdfdcc  00000001
85bdfdd0  00000000
85bdfdd4  00000000
85bdfdd8  002e0069
85bdfddc  006c0064
85bdfde0  004c006c
85bdfde4  00000000
85bdfde8  000007f0
85bdfdec  00010000
85bdfdf0  0000027f
85bdfdf4  00000000
85bdfdf8  00000000
85bdfdfc  00000000
85bdfe00  00000000
85bdfe04  00000000
85bdfe08  00001f80
85bdfe0c  0000ffff
85bdfe10  00000000
85bdfe14  00000000
85bdfe18  00000000
[...]
85bdffe4  00000000
85bdffe8  00000000
85bdffec  00000000
85bdfff0  00000000
85bdfff4  00000000
85bdfff8  00000000
85bdfffc  00000000
85be0000  ????????

User minidumps are similar here:

0:001> k
ChildEBP RetAddr 
099bfe147c90daaa ntdll!KiFastSystemCallRet
099bfe18 77e765e3 ntdll!NtReplyWaitReceivePortEx+0xc
099bff80 77e76caf rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×12a
099bff88 77e76ad1 rpcrt4!RecvLotsaCallsWrapper+0xd
099bffa8 77e76c97 rpcrt4!BaseCachedThreadRoutine+0×79
099bffb4 7c80b729 rpcrt4!ThreadStartRoutine+0×1a
099bffec 00000000 kernel32!BaseThreadStart+0×37

0:001> dd 099bfe14
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006
099bfe74  00000006 e1f79698 e39b8b60 00000000
099bfe84  fbe33c40 00000001 e5ce12f8 b279db9c

0:001> dd 099bfe14-20
099bfdf4  ???????? ???????? ???????? ????????
099bfe04  ???????? ???????? ???????? ????????
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006

As a warning here it is possible to conclude that minidumps can also reveal private information especially when ASCII or Unicode buffers are seen on raw stack data.

I was thinking how to name this pattern and Oxford Thesaurus of English suggested the following name: Abridged Dump by analogy with an abridged book.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Started testing marketing materials to increase Dublin people awareness of memory dump analysis. Here are the first exemplars of what I’m going to wear by the end of this week while commuting and in local pubs

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Blog - Binary log.

Examples: This program blogged. I’m reading its blog now.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -