Archive for July, 2012

Crash Dump Analysis Patterns (Part 180, Mac OS X)

Saturday, July 28th, 2012

This is the first pattern that emerged after applying the same pattern-driven software diagnostics methodology to Mac OS X. I had problems using GDB which is so portable that hardly has operating system support like WinDbg has. Fortunately, I found a workaround by complementing core dumps with logs and reports from OS such as crash reports and vmmap data. I call this pattern Paratext which I borrowed from the concept of an extended software trace and software narratology where it borrowed the same concept from literary interpretation (paratext). Typical examples of such pattern usage can be the list of modules with version and path info, application crash specific information, memory region names with attribution and boundaries:

// from .crash reports

0x108f99000 - 0x109044ff7 com.apple.FontBook (198.4 - 198) <7244D36E-4563-3E42-BA46-1F279D30A6CE> /Applications/Font Book.app/Contents/MacOS/Font Book

Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000

Application Specific Information:
objc[195]: garbage collection is OFF
*** error for object 0x7fd7fb818e08: incorrect checksum for freed object - object was probably modified after being freed.

// from vmmap logs

[...]
==== Writable regions for process 966
[...]
Stack 0000000101f71000-0000000101ff3000 [ 520K] rw-/rwx SM=PRV thread 1
MALLOC_LARGE 0000000103998000-00000001039b8000 [ 128K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00000001039b9000-00000001039bb000 [ 8K] rw-/rwx SM=PRV
mapped file 0000000103a05000-0000000103f32000 [ 5300K] rw-/rwx SM=COW ...box.framework/Versions/A/Resources/Extras2.rsrc
mapped file 0000000104409000-00000001046d2000 [ 2852K] rw-/rwx SM=COW /System/Library/Fonts/Helvetica.dfont
MALLOC_LARGE 0000000104f6e000-0000000104f8e000 [ 128K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_LARGE (freed) 0000000108413000-0000000108540000 [ 1204K] rw-/rwx SM=COW
MALLOC_LARGE (freed) 0000000108540000-0000000108541000 [ 4K] rw-/rwx SM=PRV
MALLOC_TINY 00007fefe0c00000-00007fefe0d00000 [ 1024K] rw-/rwx SM=COW DefaultMallocZone_0x101e6e000
MALLOC_TINY 00007fefe0d00000-00007fefe0e00000 [ 1024K] rw-/rwx SM=PRV DispatchContinuations_0x101f38000
MALLOC_TINY 00007fefe0e00000-00007fefe0f00000 [ 1024K] rw-/rwx SM=COW DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1000000-00007fefe107b000 [ 492K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe107b000-00007fefe1083000 [ 32K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1083000-00007fefe1149000 [ 792K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe1149000-00007fefe1166000 [ 116K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe1166000-00007fefe1800000 [ 6760K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1800000-00007fefe18ff000 [ 1020K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe18ff000-00007fefe1901000 [ 8K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1901000-00007fefe2000000 [ 7164K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_TINY (freed) 00007fefe2000000-00007fefe2100000 [ 1024K] rw-/rwx SM=PRV DispatchContinuations_0x101f38000
MALLOC_TINY 00007fefe2100000-00007fefe2200000 [ 1024K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
Stack 00007fff61186000-00007fff61985000 [ 8188K] rw-/rwx SM=ZER thread 0
Stack 00007fff61985000-00007fff61986000 [ 4K] rw-/rwx SM=COW
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 18, Mac OS X)

Friday, July 20th, 2012

This is a Mac OS X / GDB counterpart to Truncated Dump pattern previously described for Windows platforms:

(gdb) info threads
Cannot access memory at address 0x7fff885e9e42
4 0x00007fff885e9e42 in ?? ()
3 0x00007fff885e9e42 in ?? ()
2 0x00007fff885e9e42 in ?? ()
* 1 0x00007fff885e9e42 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0x00007fff885e9e42 in ?? ()

(gdb) disass 0x00007fff885e9e42
No function contains specified address.

(gdb) info r rsp
rsp 0x7fff67fe8a18 0x7fff67fe8a18

(gdb) x/100a 0x7fff67fe8a18
0x7fff67fe8a18: Cannot access memory at address 0x7fff67fe8a18

This often happens if there is no space to save a full core dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 77, Mac OS X)

Friday, July 20th, 2012

This is a Mac OS X / GDB counterpart to C++ Exception pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff88bd582a in __kill ()
#1 0x00007fff8c184a9c in abort ()
#2 0x00007fff852f57bc in abort_message ()
#3 0x00007fff852f2fcf in default_terminate ()
#4 0x00007fff852f3001 in safe_handler_caller ()
#5 0x00007fff852f305c in std::terminate ()
#6 0×00007fff852f4152 in __cxa_throw ()
#7 0×000000010e402be8 in bar ()
#8 0×000000010e402c99 in foo ()
#9 0×000000010e402cbb in main (argc=1, argv=0×7fff6e001b18)

The modeling application source code:

class Exception

{

    int code;

    std::string description;

 

public:

    Exception(int _code, std::string _desc) : code(_code), description(_desc) {}

};

 

void bar()

{

    throw new Exception(5, “Access Denied”);

}

 

void foo()

{

    bar();

}

 

int main(int argc, const char * argv[])

{

    foo();

    return 0;

}  

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 36, Mac OS X)

Thursday, July 19th, 2012

This is a Mac OS X / GDB counterpart to Local Buffer Overflow pattern previously described for Windows platforms. Most of the time simple mistakes in using memory and string manipulation functions are easily detected by runtime:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff8325a89f in __chk_fail ()
#3 0×00007fff8325a83e in __memcpy_chk ()

#4 0×000000010914edf3 in bar ()
#5 0×000000010914ee5e in foo ()
#6 0×000000010914ee9b in main (argc=1, argv=0×7fff68d4daf0)

This detection happens in a default optimized release version as well:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff8325a89f in __chk_fail ()
#3 0×00007fff8325a83e in __memcpy_chk ()

#4 0×000000010f59cea8 in bar [inlined] ()
#5 0×000000010f59cea8 in foo [inlined] ()
#6 0×000000010f59cea8 in main (argc=,
argv=)

The more sophisticated example which overwrites stack trace without being detected involves overwriting indirectly via a pointer to a local buffer passed to the called function. In such cases we might see incorrect and truncated stack traces:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff83285070 in __stack_chk_fail ()
#3 0×000000010524de77 in foo ()
#4 0xca4000007fff64e5 in ?? ()

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff83285070 in __stack_chk_fail ()
#3 0×0000000105ad8df7 in foo ()

Inspection of the raw stack shows ASCII-like memory values around foo symbolic reference instead of expected main and start functions:

(gdb) info r rsp
rsp 0x7fff656d79d8 0x7fff656d79d8

(gdb) x/100a 0x7fff656d79d8
0x7fff656d79d8: 0x7fff83288b6c <__abort+193> 0x0
0x7fff656d79e8: 0x0 0xffffffdf
0x7fff656d79f8: 0x7fff656d7a40 0x7fff656d7a80
0x7fff656d7a08: 0x7fff83285070 <__guard_setup> 0x6675426c61636f4c
0x7fff656d7a18: 0x7265764f726566 0x0
0x7fff656d7a28: 0x0 0x0
0x7fff656d7a38: 0x0 0x73205d343336325b
0x7fff656d7a48: 0x65766f206b636174 0x776f6c6672
0x7fff656d7a58: 0x0 0x0
0x7fff656d7a68: 0x0 0x343336326d7ab0
0x7fff656d7a78: 0x0 0x7fff656d7ab0
0x7fff656d7a88: 0x105ad8df7 0xb1887b8452358ac4
0×7fff656d7a98: 0×794d000000000000 0×6769422077654e20
0×7fff656d7aa8: 0×6666754220726567 0×7265
0×7fff656d7ab8: 0×0 0×0
0×7fff656d7ac8: 0×0 0×0
0×7fff656d7ad8: 0×0 0×0
0×7fff656d7ae8: 0×0 0×0
[…]

The modeling application source code:

void bar(char *buffer)

{

      char data[100] = “My New Bigger Buffer”

      memcpy (buffer, data, sizeof(data));

}

 

void foo()

{

    char data[10] = “My Buffer”;

    bar(data);

}

 

int main(int argc, const char * argv[])

{

    foo();

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Network Trace Analysis Patterns (Part 1)

Thursday, July 19th, 2012

After some thinking I’ve decided to apply software trace analysis pattern approach to network trace analysis which lacks a unified pattern language. Here I consider a network trace as essentially a software trace where packet headers represent software trace messages coupled with associated transmitted data:

Since we have a trace message stream formatted by a network trace visualization tool we can apply most if not all trace analysis patterns for diagnostics including software narratology for interpretation, discourse and different representations. We provide a few trivial examples here and more in subsequent parts. The first example is Discontinuity pattern:

Other similar patterns are No Activity, Truncated Trace and Time Delta. The second example is Anchor Messages:

Additional example there include Significant Event and Bifurcation Point patterns. Layered protocols are represented through Embedded Message pattern (to be described and added to the pattern list soon). Such traces can be filtered for their embedded protocol headers and therefore naturally represent Adjoint Thread pattern (for the more detailed description of adjoint threads as extension of multithreading please see the article What is an Adjoint Thread):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 78a, Mac OS X)

Wednesday, July 18th, 2012

This is a Mac OS X / GDB counterpart to Divide by Zero (user mode) pattern previously described for Windows platforms:

(gdb) bt
#0 0×000000010d3ebe9e in bar (a=1, b=0)
#1 0×000000010d3ebec3 in foo ()
#2 0×000000010d3ebeeb in main (argc=1, argv=0×7fff6cfeab18)

(gdb) x/i 0×000000010d3ebe9e
0×10d3ebe9e : idiv %esi

(gdb) info r rsi
rsi 0×0 0

The modeling application source code:

int bar(int a, int b)

{

    return a/b;

}

 

int foo()

{

    return bar(1,0);

}

 

int main(int argc, const char * argv[])

{

    return foo();

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 16b, Mac OS X)

Tuesday, July 17th, 2012

This is a Mac OS X / GDB counterpart to Stack Overflow (user mode) pattern previously described for Windows platforms:

(gdb) bt 10
#0 0x0000000105dafea8 in bar (i=0)
#1 0x0000000105dafeb9 in bar (i=262102)
#2 0x0000000105dafeb9 in bar (i=262101)
#3 0x0000000105dafeb9 in bar (i=262100)
#4 0x0000000105dafeb9 in bar (i=262099)
#5 0x0000000105dafeb9 in bar (i=262098)
#6 0x0000000105dafeb9 in bar (i=262097)
#7 0x0000000105dafeb9 in bar (i=262096)
#8 0x0000000105dafeb9 in bar (i=262095)
#9 0x0000000105dafeb9 in bar (i=262094)
(More stack frames follow...)

There are at least 262,102 frames so we don’t attempt to list them all. What we’d like to do is to get stack trace boundaries from the list of sections based on the current stack pointer address and dump the upper part of it (the stack grows from higher addresses to the lower ones) to get bottom initial stack traces:

(gdb) x $rsp
0×7fff651aeff0: 0×00000000

Because this is a stack overflow we expect RSP went out of page bounds so we expect the lowest address being 0×7fff651af000.

(gdb) maint info sections
[...]
Core file:
`/cores/core.2763', file type mach-o-le.
[...]
0x0000000105e00000->0x0000000105f00000 at 0x00035000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff619af000->0x00007fff651af000 at 0x00135000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff651af000->0×00007fff659af000 at 0×03935000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff659af000->0×00007fff659e4000 at 0×04135000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff659e4000->0×00007fff659e6000 at 0×0416a000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
[…]

(gdb) x/250a 0×00007fff659af000-2000
0×7fff659ae830: 0×0 0×1500000000
0×7fff659ae840: 0×7fff659ae860 0×105dafeb9 <bar+25>
0×7fff659ae850: 0×0 0×1400000000
0×7fff659ae860: 0×7fff659ae880 0×105dafeb9 <bar+25>
0×7fff659ae870: 0×0 0×1300000000
0×7fff659ae880: 0×7fff659ae8a0 0×105dafeb9 <bar+25>
0×7fff659ae890: 0×0 0×1200000000
0×7fff659ae8a0: 0×7fff659ae8c0 0×105dafeb9 <bar+25>
0×7fff659ae8b0: 0×0 0×1100000000
0×7fff659ae8c0: 0×7fff659ae8e0 0×105dafeb9 <bar+25>
0×7fff659ae8d0: 0×0 0×1000000000
0×7fff659ae8e0: 0×7fff659ae900 0×105dafeb9 <bar+25>
0×7fff659ae8f0: 0×0 0xf00000000
0×7fff659ae900: 0×7fff659ae920 0×105dafeb9 <bar+25>
0×7fff659ae910: 0×0 0xe00000000
0×7fff659ae920: 0×7fff659ae940 0×105dafeb9 <bar+25>
0×7fff659ae930: 0×0 0xd00000000
0×7fff659ae940: 0×7fff659ae960 0×105dafeb9 <bar+25>
0×7fff659ae950: 0×0 0xc00000000
0×7fff659ae960: 0×7fff659ae980 0×105dafeb9 <bar+25>
0×7fff659ae970: 0×0 0xb00000000
0×7fff659ae980: 0×7fff659ae9a0 0×105dafeb9 <bar+25>
0×7fff659ae990: 0×0 0xa00000000
0×7fff659ae9a0: 0×7fff659ae9c0 0×105dafeb9 <bar+25>
0×7fff659ae9b0: 0×0 0×900000000
0×7fff659ae9c0: 0×7fff659ae9e0 0×105dafeb9 <bar+25>
0×7fff659ae9d0: 0×0 0×800000000
0×7fff659ae9e0: 0×7fff659aea00 0×105dafeb9 <bar+25>
0×7fff659ae9f0: 0×0 0×700000000
0×7fff659aea00: 0×7fff659aea20 0×105dafeb9 <bar+25>
0×7fff659aea10: 0×0 0×600000000
0×7fff659aea20: 0×7fff659aea40 0×105dafeb9 <bar+25>
0×7fff659aea30: 0×0 0×5659b9fe0
0×7fff659aea40: 0×7fff659aea60 0×105dafeb9 <bar+25
0×7fff659aea50: 0×7fff659aea70 0×4659bd31f
0×7fff659aea60: 0×7fff659aea80 0×105dafeb9 <bar+25>
0×7fff659aea70: 0×7fff659aeaf0 0×3659b031a
0×7fff659aea80: 0×7fff659aeaa0 0×105dafeb9 <bar+25>
0×7fff659aea90: 0×7fff659af5c0 0×200000000
0×7fff659aeaa0: 0×7fff659aeac0 0×105dafeb9 <bar+25>
0×7fff659aeab0: 0×100000000 0×1659aeb18
0×7fff659aeac0: 0×7fff659aead0 0×105dafece <foo+14>
0×7fff659aead0: 0×7fff659aeaf0 0×105dafeeb <main+27>
0×7fff659aeae0: 0×7fff659aeb18 0×1
—Type to continue, or q to quit—
0×7fff659aeaf0: 0×7fff659aeb08 0×105dafe94 <start+52>
0×7fff659aeb00: 0×0 0×0
[…]
0×7fff659aeff0: 0×3139336561303363 0×316235

Interesting if we set the lowest frame down and try to get register info GDB core dumps:

(gdb) frame 262102
#262102 0x0000000105dafeb9 in bar (i=1)
13 bar(i+1);
(gdb) info r
Segmentation fault: 11 (core dumped)

Looking its core dump show that it also experienced stack overflow:

(gdb) bt
#0 0x00007fff8c1bacf0 in __sfvwrite ()
#1 0x00007fff8c189947 in __vfprintf ()
#2 0x00007fff8c184edb in vsnprintf_l ()
#3 0x00007fff8c1566be in __sprintf_chk ()
#4 0x000000010bd14d15 in print_displacement ()
#5 0x000000010bd10ddf in OP_E ()
#6 0x000000010bd13f9b in print_insn ()
#7 0x000000010bc164ce in length_of_this_instruction ()
#8 0x000000010bc9e296 in x86_analyze_prologue ()
#9 0x000000010bc9f1f3 in x86_frame_prev_register ()
#10 0x000000010bc91d70 in frame_register_unwind ()
#11 0x000000010bc92015 in frame_unwind_register ()
#12 0x000000010bc91d70 in frame_register_unwind ()
#13 0x000000010bc92015 in frame_unwind_register ()
#14 0x000000010bc91d70 in frame_register_unwind ()
#15 0x000000010bc92015 in frame_unwind_register ()
#16 0x000000010bc91d70 in frame_register_unwind ()
#17 0x000000010bc92015 in frame_unwind_register ()
#18 0x000000010bc91d70 in frame_register_unwind ()
#19 0x000000010bc92015 in frame_unwind_register ()
#20 0x000000010bc91d70 in frame_register_unwind ()
#21 0x000000010bc92015 in frame_unwind_register ()
#22 0x000000010bc91d70 in frame_register_unwind ()
#23 0x000000010bc92015 in frame_unwind_register ()
#24 0x000000010bc91d70 in frame_register_unwind ()
#25 0x000000010bc92015 in frame_unwind_register ()
#26 0x000000010bc91d70 in frame_register_unwind ()
#27 0x000000010bc92015 in frame_unwind_register ()
#28 0x000000010bc91d70 in frame_register_unwind ()
#29 0x000000010bc92015 in frame_unwind_register ()
#30 0x000000010bc91d70 in frame_register_unwind ()
#31 0x000000010bc92015 in frame_unwind_register ()
#32 0x000000010bc91d70 in frame_register_unwind ()
#33 0x000000010bc92015 in frame_unwind_register ()
#34 0x000000010bc91d70 in frame_register_unwind ()
#35 0x000000010bc92015 in frame_unwind_register ()
#36 0x000000010bc91d70 in frame_register_unwind ()
#37 0x000000010bc92015 in frame_unwind_register ()
#38 0x000000010bc91d70 in frame_register_unwind ()
#39 0x000000010bc92015 in frame_unwind_register ()
#40 0x000000010bc91d70 in frame_register_unwind ()
#41 0x000000010bc92015 in frame_unwind_register ()
#42 0x000000010bc91d70 in frame_register_unwind ()
#43 0x000000010bc92015 in frame_unwind_register ()

The source code of our modeling application:

void bar(int i)

{

    bar(i+1);

}

 

void foo()

{

    bar(1);

}

 

int main(int argc, const char * argv[])

{

    foo();

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Module Patterns

Sunday, July 15th, 2012

A page to reference all different kinds of module and component related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 179)

Sunday, July 15th, 2012

When looking at the module list (lmv), searching for modules (.imgscan) or examining the particular module (!address, !dh) we may notice one of them as Deviant Module. The deviation may be in (but not limited to as anything is possible):

- suspicious module name

- suspicious protection

- suspicious module load address

0:005> .imgscan
MZ at 00040000, prot 00000040, type 00020000 - size 1d000
MZ at 00340000, prot 00000002, type 01000000 - size 9c000
Name: iexplore.exe
MZ at 02250000, prot 00000002, type 00040000 - size 2000
MZ at 023b0000, prot 00000002, type 01000000 - size b000
Name: msimtf.dll
MZ at 03f80000, prot 00000002, type 00040000 - size 2000
MZ at 10000000, prot 00000004, type 00020000 - size 5000
Name: screens_dll.dll
MZ at 16080000, prot 00000002, type 01000000 - size 25000
Name: mdnsNSP.dll
MZ at 6ab50000, prot 00000002, type 01000000 - size 26000
Name: DSSENH.dll
MZ at 6b030000, prot 00000002, type 01000000 - size 5b0000
Name: MSHTML.dll
MZ at 6ba10000, prot 00000002, type 01000000 - size b4000
Name: JSCRIPT.dll
MZ at 6cec0000, prot 00000002, type 01000000 - size 1b000
Name: CRYPTNET.dll
MZ at 6d260000, prot 00000002, type 01000000 - size e000
Name: PNGFILTER.DLL
MZ at 6d2f0000, prot 00000002, type 01000000 - size 29000
Name: msls31.dll
MZ at 6d700000, prot 00000002, type 01000000 - size 30000
Name: MLANG.dll
MZ at 6d740000, prot 00000002, type 01000000 - size 4d000
Name: SSV.DLL
MZ at 6d7b0000, prot 00000002, type 01000000 - size c000
Name: ImgUtil.dll
MZ at 6ddb0000, prot 00000002, type 01000000 - size 2f000
Name: iepeers.DLL
MZ at 6df20000, prot 00000002, type 01000000 - size 33000
Name: IEShims.dll
MZ at 6eb80000, prot 00000002, type 01000000 - size a94000
Name: IEFRAME.dll
MZ at 703b0000, prot 00000002, type 01000000 - size 53000
Name: SWEEPRX.dll
MZ at 70740000, prot 00000002, type 01000000 - size 40000
Name: SWEEPRX.dll
MZ at 725a0000, prot 00000002, type 01000000 - size 12000
Name: PNRPNSP.dll
MZ at 725d0000, prot 00000002, type 01000000 - size 8000
Name: WINRNR.dll
MZ at 725e0000, prot 00000002, type 01000000 - size 136000
Name: MSXML3.dll
MZ at 72720000, prot 00000002, type 01000000 - size c000
Name: wshbth.dll
MZ at 72730000, prot 00000002, type 01000000 - size f000
Name: NAPINSP.dll
MZ at 72890000, prot 00000002, type 01000000 - size 6000
Name: SensApi.dll
MZ at 72ec0000, prot 00000002, type 01000000 - size 42000
Name: WINSPOOL.DRV
MZ at 734b0000, prot 00000002, type 01000000 - size 6000
Name: rasadhlp.dll
MZ at 736b0000, prot 00000002, type 01000000 - size 85000
Name: COMCTL32.dll
MZ at 73ac0000, prot 00000002, type 01000000 - size 7000
Name: MIDIMAP.dll
MZ at 73ae0000, prot 00000002, type 01000000 - size 14000
Name: MSACM32.dll
MZ at 73b00000, prot 00000002, type 01000000 - size 66000
Name: audioeng.dll
MZ at 73c30000, prot 00000002, type 01000000 - size 9000
Name: MSACM32.DRV
MZ at 73c60000, prot 00000002, type 01000000 - size 21000
Name: AudioSes.DLL
MZ at 73c90000, prot 00000002, type 01000000 - size 2f000
Name: WINMMDRV.dll
MZ at 74290000, prot 00000002, type 01000000 - size bb000
Name: PROPSYS.dll
MZ at 74390000, prot 00000002, type 01000000 - size f000
Name: nlaapi.dll
MZ at 743a0000, prot 00000002, type 01000000 - size 4000
Name: ksuser.dll
MZ at 74430000, prot 00000002, type 01000000 - size 15000
Name: Cabinet.dll
MZ at 74450000, prot 00000002, type 01000000 - size 3d000
Name: OLEACC.dll
MZ at 74490000, prot 00000002, type 01000000 - size 1ab000
Name: gdiplus.dll
MZ at 74640000, prot 00000002, type 01000000 - size 28000
Name: MMDevAPI.DLL
MZ at 74670000, prot 00000002, type 01000000 - size 32000
Name: WINMM.dll
MZ at 746b0000, prot 00000002, type 01000000 - size 31000
Name: TAPI32.dll
MZ at 749e0000, prot 00000002, type 01000000 - size 19e000
Name: COMCTL32.dll
MZ at 74b80000, prot 00000002, type 01000000 - size 7000
Name: AVRT.dll
MZ at 74ba0000, prot 00000002, type 01000000 - size 4a000
Name: RASAPI32.dll
MZ at 74ce0000, prot 00000002, type 01000000 - size 3f000
Name: UxTheme.dll
MZ at 74de0000, prot 00000002, type 01000000 - size 2d000
Name: WINTRUST.dll
MZ at 74ea0000, prot 00000002, type 01000000 - size 14000
Name: rasman.dll
MZ at 74f70000, prot 00000002, type 01000000 - size c000
Name: rtutils.dll
MZ at 74f80000, prot 00000002, type 01000000 - size 5000
Name: WSHTCPIP.dll
MZ at 74fb0000, prot 00000002, type 01000000 - size 21000
Name: NTMARTA.dll
MZ at 75010000, prot 00000002, type 01000000 - size 3b000
Name: RSAENH.dll
MZ at 75050000, prot 00000002, type 01000000 - size 5000
Name: MSIMG32.dll
MZ at 75060000, prot 00000002, type 01000000 - size 15000
Name: GPAPI.dll
MZ at 750a0000, prot 00000002, type 01000000 - size 46000
Name: SCHANNEL.dll
MZ at 752b0000, prot 00000002, type 01000000 - size 3b000
Name: MSWSOCK.dll
MZ at 75370000, prot 00000002, type 01000000 - size 45000
Name: bcrypt.dll
MZ at 753f0000, prot 00000002, type 01000000 - size 5000
Name: WSHIP6.dll
MZ at 75400000, prot 00000002, type 01000000 - size 8000
Name: VERSION.dll
MZ at 75420000, prot 00000002, type 01000000 - size 7000
Name: CREDSSP.dll
MZ at 75430000, prot 00000002, type 01000000 - size 35000
Name: ncrypt.dll
MZ at 75480000, prot 00000002, type 01000000 - size 22000
Name: dhcpcsvc6.DLL
MZ at 754b0000, prot 00000002, type 01000000 - size 7000
Name: WINNSI.DLL
MZ at 754c0000, prot 00000002, type 01000000 - size 35000
Name: dhcpcsvc.DLL
MZ at 75500000, prot 00000002, type 01000000 - size 19000
Name: IPHLPAPI.DLL
MZ at 75590000, prot 00000002, type 01000000 - size 3a000
Name: slc.dll
MZ at 755d0000, prot 00000002, type 01000000 - size f2000
Name: CRYPT32.dll
MZ at 75740000, prot 00000002, type 01000000 - size 12000
Name: MSASN1.dll
MZ at 75760000, prot 00000002, type 01000000 - size 11000
Name: SAMLIB.dll
MZ at 75780000, prot 00000002, type 01000000 - size 76000
Name: NETAPI32.dll
MZ at 75800000, prot 00000002, type 01000000 - size 2c000
Name: DNSAPI.dll
MZ at 75a70000, prot 00000002, type 01000000 - size 5f000
Name: sxs.dll
MZ at 75ad0000, prot 00000002, type 01000000 - size 2c000
Name: apphelp.dll
MZ at 75b30000, prot 00000002, type 01000000 - size 14000
Name: Secur32.dll
MZ at 75b50000, prot 00000002, type 01000000 - size 1e000
Name: USERENV.dll
MZ at 75c90000, prot 00000002, type 01000000 - size 7000
Name: PSAPI.DLL
MZ at 75ca0000, prot 00000002, type 01000000 - size c3000
Name: RPCRT4.dll
MZ at 75d70000, prot 00000002, type 01000000 - size 73000
Name: COMDLG32.dll
MZ at 75df0000, prot 00000002, type 01000000 - size 9000
Name: LPK.dll
MZ at 75e00000, prot 00000002, type 01000000 - size dc000
Name: KERNEL32.dll
MZ at 75ee0000, prot 00000002, type 01000000 - size aa000
Name: msvcrt.dll
MZ at 75f90000, prot 00000002, type 01000000 - size 1e8000
Name: iertutil.dll
MZ at 76180000, prot 00000002, type 01000000 - size 29000
Name: imagehlp.dll
MZ at 761b0000, prot 00000002, type 01000000 - size 6000
Name: NSI.dll
MZ at 761c0000, prot 00000002, type 01000000 - size 84000
Name: CLBCatQ.DLL
MZ at 76250000, prot 00000002, type 01000000 - size 49000
Name: WLDAP32.dll
MZ at 762a0000, prot 00000002, type 01000000 - size c6000
Name: ADVAPI32.dll
MZ at 76370000, prot 00000002, type 01000000 - size 4b000
Name: GDI32.dll
MZ at 763c0000, prot 00000002, type 01000000 - size 59000
Name: SHLWAPI.dll
MZ at 76420000, prot 00000002, type 01000000 - size e6000
Name: WININET.dll
MZ at 76510000, prot 00000002, type 01000000 - size b10000
Name: SHELL32.dll
MZ at 77020000, prot 00000002, type 01000000 - size 145000
Name: ole32.dll
MZ at 77170000, prot 00000002, type 01000000 - size 7d000
Name: USP10.dll
MZ at 771f0000, prot 00000002, type 01000000 - size 8d000
Name: OLEAUT32.dll
MZ at 77280000, prot 00000002, type 01000000 - size 18a000
Name: SETUPAPI.dll
MZ at 77410000, prot 00000002, type 01000000 - size 9d000
Name: USER32.dll
MZ at 774b0000, prot 00000002, type 01000000 - size 133000
Name: urlmon.dll
MZ at 775f0000, prot 00000002, type 01000000 - size 127000
Name: ntdll.dll
MZ at 77720000, prot 00000002, type 01000000 - size 3000
Name: Normaliz.dll
MZ at 77730000, prot 00000002, type 01000000 - size 2d000
Name: WS2_32.dll
MZ at 77760000, prot 00000002, type 01000000 - size 1e000
Name: IMM32.dll
MZ at 77780000, prot 00000002, type 01000000 - size c8000
Name: MSCTF.dll
MZ at 7c340000, prot 00000002, type 01000000 - size 56000
Name: MSVCR71.dll

0:005> !address 00040000
Usage:                  <unclassified>
Allocation Base:        00040000
Base Address:           00040000
End Address:            0005d000
Region Size:            0001d000
Type:                   00020000 MEM_PRIVATE
State:                  00001000 MEM_COMMIT
Protect:                00000040 PAGE_EXECUTE_READWRITE

0:005> !address 10000000
Usage:                  <unclassified>
Allocation Base:        10000000
Base Address:           10000000
End Address:            10001000
Region Size:            00001000
Type:                   00020000 MEM_PRIVATE
State:                  00001000 MEM_COMMIT
Protect:                00000004 PAGE_READWRITE

- suspicious text inside

See this case study for an example.

- suspicious import table (screen grabbing) or its absence (dynamic imports)

0:005> !dh 10000000
[...]
2330 [      50] address [size] of Export Directory
20E0 [      78] address [size] of Import Directory
0 [       0] address [size] of Resource Directory
0 [       0] address [size] of Exception Directory
0 [       0] address [size] of Security Directory
4000 [      34] address [size] of Base Relocation Directory
2060 [      1C] address [size] of Debug Directory
0 [       0] address [size] of Description Directory
0 [       0] address [size] of Special Directory
0 [       0] address [size] of Thread Storage Directory
0 [       0] address [size] of Load Configuration Directory
0 [       0] address [size] of Bound Import Directory
2000 [      58] address [size] of Import Address Table Directory
0 [       0] address [size] of Delay Import Directory
0 [       0] address [size] of COR20 Header Directory
0 [       0] address [size] of Reserved Directory
[…]

0:005> dps 10000000+2000 10000000+2000+58
10002000  76376101 gdi32!CreateCompatibleDC
10002004  763793d6 gdi32!StretchBlt
10002008  76377461 gdi32!CreateDIBSection
1000200c  763762a0 gdi32!SelectObject

10002010  00000000
10002014  75e4a411 kernel32!lstrcmpW
10002018  75e440aa kernel32!VirtualFree
1000201c  75e4ad55 kernel32!VirtualAlloc
10002020  00000000
10002024  77429ced user32!ReleaseDC
10002028  77423ba7 user32!NtUserGetWindowDC
1000202c  77430e21 user32!GetWindowRect

10002030  00000000
10002034  744a75e9 GdiPlus!GdiplusStartup
10002038  744976dd GdiPlus!GdipSaveImageToStream
1000203c  744cdd38 GdiPlus!GdipGetImageEncodersSize
10002040  744971cf GdiPlus!GdipDisposeImage
10002044  744a8591 GdiPlus!GdipCreateBitmapFromHBITMAP
10002048  744cdbae GdiPlus!GdipGetImageEncoders

1000204c  00000000
10002050  7707d51b ole32!CreateStreamOnHGlobal
10002054  00000000
10002058  00000000

0:000> !dh 012a0000
[...]
0 [       0] address [size] of Export Directory
0 [       0] address [size] of Import Directory
0 [       0] address [size] of Resource Directory
0 [       0] address [size] of Exception Directory
0 [       0] address [size] of Security Directory
8000 [      FC] address [size] of Base Relocation Directory
4000 [      1C] address [size] of Debug Directory
0 [       0] address [size] of Description Directory
0 [       0] address [size] of Special Directory
0 [       0] address [size] of Thread Storage Directory
0 [       0] address [size] of Load Configuration Directory
0 [       0] address [size] of Bound Import Directory
0 [       0] address [size] of Import Address Table Directory
0 [       0] address [size] of Delay Import Directory
0 [       0] address [size] of COR20 Header Directory
0 [       0] address [size] of Reserved Directory
[…]

- suspicious path names

Age: 7, Pdb: d:\work\BekConnekt\Client_src_code_New\Release\Blackjoe_new.pdb

Debug Directories(1)
Type Size Address Pointer
cv 46 2094 894 Format: RSDS, guid, 1, C:\MyWork\screens_dll\Release\screens_dll.pdb

- suspicious image path (although could be just dynamic code generation for .NET assemblies)

- uninitialized image resources

0:002> lmv m C6DC
start    end        module name
012a0000 012a9000   C6DC     C (no symbols)
Loaded symbol image file: C6DC.tmp
Image path: C:\Users\User\AppData\Local\Temp\C6DC.tmp
Image name: C6DC.tmp
Timestamp:        Sun May 30 20:18:32 2010 (4C02BA08)
CheckSum:         00000000
ImageSize:        00009000
File version:     0.0.0.0
Product version:  0.0.0.0
File flags:       0 (Mask 0)
File OS:          0 Unknown Base
File type:        0.0 Unknown
File date:        00000000.00000000

Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

- suspicious (small) image size

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Software Diagnostics Services

Friday, July 13th, 2012

For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:

The First Comprehensive Software Diagnostics Service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Software Trace Diagrams (STDiagrams)

Tuesday, July 3rd, 2012

When depicting trace analysis patterns I used two-dimensional diagrams based on CDF (ETW) traces such as this one for a bifurcation point:

While working today on a new pattern I needed a new expressive way to graphically illustrate the same idea of trace bifurcation points but without too much drawing. Traces from particle chambers and scattering diagrams came to my imagination after draw the first few diagrams illustrating bifurcation points:

Time directional arrow end can be omitted:

Trace variation after a bifurcation point can be indicated by angle partition:

The case when a variation also happens before is illustrated on this diagram:

and the case with several bifurcations:

Are N-bifurcations like on the diagram below possible?

Yes, they are, if the course of execution depends on some non-binary trace message parameter such as a loaded module that implements a required interface differently.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Architecture of Process Memory Dump Capture Done Right

Monday, July 2nd, 2012

Sometimes I get requests to review application memory dump capture design. Of course, such requests usually come only when such designs don’t work or there are problems with loading saved crash dumps. The common blueprint of such architectures is a top level exception handler that use some API do capture and save process memory state. However, such designs forget why separate processed were introduced in the first place: to guard process memory space of different unrelated tasks (for related tasks there are threads). The data of the module (and its thread state) that does process memory capture may also be corrupt. The right design would be to show a message box with an information on how to use external process memory dumper such as Task Manager. If we need an automation then the right thing is to rely on WER features. Let separate processes do their work in separate spaces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -