Archive for the ‘x64 Mac OS X’ Category

Software Disruption Patterns (Part 1)

Saturday, January 12th, 2013

These are not really testing patterns but patterns of a user or program behaviour resulting in abnormalities such as colored screens (blue and grey), crash dumps and other support artefacts including performance alerts and UI problem patterns. The first such pattern is called Space Precondition. It is often a case that a process checks for free disk space before proceeding, for example, with updates. This precondition may be violated by a user filling disk after the check but before or during installation. Such was the case with our recent Mac OS X update where we did some copying in the background while the update was downloaded that resulted in insufficient space after reboot, then a grey screen after that and subsequent disk corruption and finally reinstalling OS and loss of several hours better spent with other pattern categories.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 180, Mac OS X)

Saturday, July 28th, 2012

This is the first pattern that emerged after applying the same pattern-driven software diagnostics methodology to Mac OS X. I had problems using GDB which is so portable that hardly has operating system support like WinDbg has. Fortunately, I found a workaround by complementing core dumps with logs and reports from OS such as crash reports and vmmap data. I call this pattern Paratext which I borrowed from the concept of an extended software trace and software narratology where it borrowed the same concept from literary interpretation (paratext). Typical examples of such pattern usage can be the list of modules with version and path info, application crash specific information, memory region names with attribution and boundaries:

// from .crash reports

0x108f99000 - 0x109044ff7 com.apple.FontBook (198.4 - 198) <7244D36E-4563-3E42-BA46-1F279D30A6CE> /Applications/Font Book.app/Contents/MacOS/Font Book

Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000

Application Specific Information:
objc[195]: garbage collection is OFF
*** error for object 0x7fd7fb818e08: incorrect checksum for freed object - object was probably modified after being freed.

// from vmmap logs

[...]
==== Writable regions for process 966
[...]
Stack 0000000101f71000-0000000101ff3000 [ 520K] rw-/rwx SM=PRV thread 1
MALLOC_LARGE 0000000103998000-00000001039b8000 [ 128K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00000001039b9000-00000001039bb000 [ 8K] rw-/rwx SM=PRV
mapped file 0000000103a05000-0000000103f32000 [ 5300K] rw-/rwx SM=COW ...box.framework/Versions/A/Resources/Extras2.rsrc
mapped file 0000000104409000-00000001046d2000 [ 2852K] rw-/rwx SM=COW /System/Library/Fonts/Helvetica.dfont
MALLOC_LARGE 0000000104f6e000-0000000104f8e000 [ 128K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_LARGE (freed) 0000000108413000-0000000108540000 [ 1204K] rw-/rwx SM=COW
MALLOC_LARGE (freed) 0000000108540000-0000000108541000 [ 4K] rw-/rwx SM=PRV
MALLOC_TINY 00007fefe0c00000-00007fefe0d00000 [ 1024K] rw-/rwx SM=COW DefaultMallocZone_0x101e6e000
MALLOC_TINY 00007fefe0d00000-00007fefe0e00000 [ 1024K] rw-/rwx SM=PRV DispatchContinuations_0x101f38000
MALLOC_TINY 00007fefe0e00000-00007fefe0f00000 [ 1024K] rw-/rwx SM=COW DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1000000-00007fefe107b000 [ 492K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe107b000-00007fefe1083000 [ 32K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1083000-00007fefe1149000 [ 792K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe1149000-00007fefe1166000 [ 116K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe1166000-00007fefe1800000 [ 6760K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1800000-00007fefe18ff000 [ 1020K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe18ff000-00007fefe1901000 [ 8K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1901000-00007fefe2000000 [ 7164K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_TINY (freed) 00007fefe2000000-00007fefe2100000 [ 1024K] rw-/rwx SM=PRV DispatchContinuations_0x101f38000
MALLOC_TINY 00007fefe2100000-00007fefe2200000 [ 1024K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
Stack 00007fff61186000-00007fff61985000 [ 8188K] rw-/rwx SM=ZER thread 0
Stack 00007fff61985000-00007fff61986000 [ 4K] rw-/rwx SM=COW
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 18, Mac OS X)

Friday, July 20th, 2012

This is a Mac OS X / GDB counterpart to Truncated Dump pattern previously described for Windows platforms:

(gdb) info threads
Cannot access memory at address 0x7fff885e9e42
4 0x00007fff885e9e42 in ?? ()
3 0x00007fff885e9e42 in ?? ()
2 0x00007fff885e9e42 in ?? ()
* 1 0x00007fff885e9e42 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0x00007fff885e9e42 in ?? ()

(gdb) disass 0x00007fff885e9e42
No function contains specified address.

(gdb) info r rsp
rsp 0x7fff67fe8a18 0x7fff67fe8a18

(gdb) x/100a 0x7fff67fe8a18
0x7fff67fe8a18: Cannot access memory at address 0x7fff67fe8a18

This often happens if there is no space to save a full core dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 77, Mac OS X)

Friday, July 20th, 2012

This is a Mac OS X / GDB counterpart to C++ Exception pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff88bd582a in __kill ()
#1 0x00007fff8c184a9c in abort ()
#2 0x00007fff852f57bc in abort_message ()
#3 0x00007fff852f2fcf in default_terminate ()
#4 0x00007fff852f3001 in safe_handler_caller ()
#5 0x00007fff852f305c in std::terminate ()
#6 0×00007fff852f4152 in __cxa_throw ()
#7 0×000000010e402be8 in bar ()
#8 0×000000010e402c99 in foo ()
#9 0×000000010e402cbb in main (argc=1, argv=0×7fff6e001b18)

The modeling application source code:

class Exception

{

    int code;

    std::string description;

 

public:

    Exception(int _code, std::string _desc) : code(_code), description(_desc) {}

};

 

void bar()

{

    throw new Exception(5, “Access Denied”);

}

 

void foo()

{

    bar();

}

 

int main(int argc, const char * argv[])

{

    foo();

    return 0;

}  

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 36, Mac OS X)

Thursday, July 19th, 2012

This is a Mac OS X / GDB counterpart to Local Buffer Overflow pattern previously described for Windows platforms. Most of the time simple mistakes in using memory and string manipulation functions are easily detected by runtime:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff8325a89f in __chk_fail ()
#3 0×00007fff8325a83e in __memcpy_chk ()

#4 0×000000010914edf3 in bar ()
#5 0×000000010914ee5e in foo ()
#6 0×000000010914ee9b in main (argc=1, argv=0×7fff68d4daf0)

This detection happens in a default optimized release version as well:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff8325a89f in __chk_fail ()
#3 0×00007fff8325a83e in __memcpy_chk ()

#4 0×000000010f59cea8 in bar [inlined] ()
#5 0×000000010f59cea8 in foo [inlined] ()
#6 0×000000010f59cea8 in main (argc=,
argv=)

The more sophisticated example which overwrites stack trace without being detected involves overwriting indirectly via a pointer to a local buffer passed to the called function. In such cases we might see incorrect and truncated stack traces:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff83285070 in __stack_chk_fail ()
#3 0×000000010524de77 in foo ()
#4 0xca4000007fff64e5 in ?? ()

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff83285070 in __stack_chk_fail ()
#3 0×0000000105ad8df7 in foo ()

Inspection of the raw stack shows ASCII-like memory values around foo symbolic reference instead of expected main and start functions:

(gdb) info r rsp
rsp 0x7fff656d79d8 0x7fff656d79d8

(gdb) x/100a 0x7fff656d79d8
0x7fff656d79d8: 0x7fff83288b6c <__abort+193> 0x0
0x7fff656d79e8: 0x0 0xffffffdf
0x7fff656d79f8: 0x7fff656d7a40 0x7fff656d7a80
0x7fff656d7a08: 0x7fff83285070 <__guard_setup> 0x6675426c61636f4c
0x7fff656d7a18: 0x7265764f726566 0x0
0x7fff656d7a28: 0x0 0x0
0x7fff656d7a38: 0x0 0x73205d343336325b
0x7fff656d7a48: 0x65766f206b636174 0x776f6c6672
0x7fff656d7a58: 0x0 0x0
0x7fff656d7a68: 0x0 0x343336326d7ab0
0x7fff656d7a78: 0x0 0x7fff656d7ab0
0x7fff656d7a88: 0x105ad8df7 0xb1887b8452358ac4
0×7fff656d7a98: 0×794d000000000000 0×6769422077654e20
0×7fff656d7aa8: 0×6666754220726567 0×7265
0×7fff656d7ab8: 0×0 0×0
0×7fff656d7ac8: 0×0 0×0
0×7fff656d7ad8: 0×0 0×0
0×7fff656d7ae8: 0×0 0×0
[…]

The modeling application source code:

void bar(char *buffer)

{

      char data[100] = “My New Bigger Buffer”

      memcpy (buffer, data, sizeof(data));

}

 

void foo()

{

    char data[10] = “My Buffer”;

    bar(data);

}

 

int main(int argc, const char * argv[])

{

    foo();

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 78a, Mac OS X)

Wednesday, July 18th, 2012

This is a Mac OS X / GDB counterpart to Divide by Zero (user mode) pattern previously described for Windows platforms:

(gdb) bt
#0 0×000000010d3ebe9e in bar (a=1, b=0)
#1 0×000000010d3ebec3 in foo ()
#2 0×000000010d3ebeeb in main (argc=1, argv=0×7fff6cfeab18)

(gdb) x/i 0×000000010d3ebe9e
0×10d3ebe9e : idiv %esi

(gdb) info r rsi
rsi 0×0 0

The modeling application source code:

int bar(int a, int b)

{

    return a/b;

}

 

int foo()

{

    return bar(1,0);

}

 

int main(int argc, const char * argv[])

{

    return foo();

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 16b, Mac OS X)

Tuesday, July 17th, 2012

This is a Mac OS X / GDB counterpart to Stack Overflow (user mode) pattern previously described for Windows platforms:

(gdb) bt 10
#0 0x0000000105dafea8 in bar (i=0)
#1 0x0000000105dafeb9 in bar (i=262102)
#2 0x0000000105dafeb9 in bar (i=262101)
#3 0x0000000105dafeb9 in bar (i=262100)
#4 0x0000000105dafeb9 in bar (i=262099)
#5 0x0000000105dafeb9 in bar (i=262098)
#6 0x0000000105dafeb9 in bar (i=262097)
#7 0x0000000105dafeb9 in bar (i=262096)
#8 0x0000000105dafeb9 in bar (i=262095)
#9 0x0000000105dafeb9 in bar (i=262094)
(More stack frames follow...)

There are at least 262,102 frames so we don’t attempt to list them all. What we’d like to do is to get stack trace boundaries from the list of sections based on the current stack pointer address and dump the upper part of it (the stack grows from higher addresses to the lower ones) to get bottom initial stack traces:

(gdb) x $rsp
0×7fff651aeff0: 0×00000000

Because this is a stack overflow we expect RSP went out of page bounds so we expect the lowest address being 0×7fff651af000.

(gdb) maint info sections
[...]
Core file:
`/cores/core.2763', file type mach-o-le.
[...]
0x0000000105e00000->0x0000000105f00000 at 0x00035000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff619af000->0x00007fff651af000 at 0x00135000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff651af000->0×00007fff659af000 at 0×03935000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff659af000->0×00007fff659e4000 at 0×04135000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff659e4000->0×00007fff659e6000 at 0×0416a000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
[…]

(gdb) x/250a 0×00007fff659af000-2000
0×7fff659ae830: 0×0 0×1500000000
0×7fff659ae840: 0×7fff659ae860 0×105dafeb9 <bar+25>
0×7fff659ae850: 0×0 0×1400000000
0×7fff659ae860: 0×7fff659ae880 0×105dafeb9 <bar+25>
0×7fff659ae870: 0×0 0×1300000000
0×7fff659ae880: 0×7fff659ae8a0 0×105dafeb9 <bar+25>
0×7fff659ae890: 0×0 0×1200000000
0×7fff659ae8a0: 0×7fff659ae8c0 0×105dafeb9 <bar+25>
0×7fff659ae8b0: 0×0 0×1100000000
0×7fff659ae8c0: 0×7fff659ae8e0 0×105dafeb9 <bar+25>
0×7fff659ae8d0: 0×0 0×1000000000
0×7fff659ae8e0: 0×7fff659ae900 0×105dafeb9 <bar+25>
0×7fff659ae8f0: 0×0 0xf00000000
0×7fff659ae900: 0×7fff659ae920 0×105dafeb9 <bar+25>
0×7fff659ae910: 0×0 0xe00000000
0×7fff659ae920: 0×7fff659ae940 0×105dafeb9 <bar+25>
0×7fff659ae930: 0×0 0xd00000000
0×7fff659ae940: 0×7fff659ae960 0×105dafeb9 <bar+25>
0×7fff659ae950: 0×0 0xc00000000
0×7fff659ae960: 0×7fff659ae980 0×105dafeb9 <bar+25>
0×7fff659ae970: 0×0 0xb00000000
0×7fff659ae980: 0×7fff659ae9a0 0×105dafeb9 <bar+25>
0×7fff659ae990: 0×0 0xa00000000
0×7fff659ae9a0: 0×7fff659ae9c0 0×105dafeb9 <bar+25>
0×7fff659ae9b0: 0×0 0×900000000
0×7fff659ae9c0: 0×7fff659ae9e0 0×105dafeb9 <bar+25>
0×7fff659ae9d0: 0×0 0×800000000
0×7fff659ae9e0: 0×7fff659aea00 0×105dafeb9 <bar+25>
0×7fff659ae9f0: 0×0 0×700000000
0×7fff659aea00: 0×7fff659aea20 0×105dafeb9 <bar+25>
0×7fff659aea10: 0×0 0×600000000
0×7fff659aea20: 0×7fff659aea40 0×105dafeb9 <bar+25>
0×7fff659aea30: 0×0 0×5659b9fe0
0×7fff659aea40: 0×7fff659aea60 0×105dafeb9 <bar+25
0×7fff659aea50: 0×7fff659aea70 0×4659bd31f
0×7fff659aea60: 0×7fff659aea80 0×105dafeb9 <bar+25>
0×7fff659aea70: 0×7fff659aeaf0 0×3659b031a
0×7fff659aea80: 0×7fff659aeaa0 0×105dafeb9 <bar+25>
0×7fff659aea90: 0×7fff659af5c0 0×200000000
0×7fff659aeaa0: 0×7fff659aeac0 0×105dafeb9 <bar+25>
0×7fff659aeab0: 0×100000000 0×1659aeb18
0×7fff659aeac0: 0×7fff659aead0 0×105dafece <foo+14>
0×7fff659aead0: 0×7fff659aeaf0 0×105dafeeb <main+27>
0×7fff659aeae0: 0×7fff659aeb18 0×1
—Type to continue, or q to quit—
0×7fff659aeaf0: 0×7fff659aeb08 0×105dafe94 <start+52>
0×7fff659aeb00: 0×0 0×0
[…]
0×7fff659aeff0: 0×3139336561303363 0×316235

Interesting if we set the lowest frame down and try to get register info GDB core dumps:

(gdb) frame 262102
#262102 0x0000000105dafeb9 in bar (i=1)
13 bar(i+1);
(gdb) info r
Segmentation fault: 11 (core dumped)

Looking its core dump show that it also experienced stack overflow:

(gdb) bt
#0 0x00007fff8c1bacf0 in __sfvwrite ()
#1 0x00007fff8c189947 in __vfprintf ()
#2 0x00007fff8c184edb in vsnprintf_l ()
#3 0x00007fff8c1566be in __sprintf_chk ()
#4 0x000000010bd14d15 in print_displacement ()
#5 0x000000010bd10ddf in OP_E ()
#6 0x000000010bd13f9b in print_insn ()
#7 0x000000010bc164ce in length_of_this_instruction ()
#8 0x000000010bc9e296 in x86_analyze_prologue ()
#9 0x000000010bc9f1f3 in x86_frame_prev_register ()
#10 0x000000010bc91d70 in frame_register_unwind ()
#11 0x000000010bc92015 in frame_unwind_register ()
#12 0x000000010bc91d70 in frame_register_unwind ()
#13 0x000000010bc92015 in frame_unwind_register ()
#14 0x000000010bc91d70 in frame_register_unwind ()
#15 0x000000010bc92015 in frame_unwind_register ()
#16 0x000000010bc91d70 in frame_register_unwind ()
#17 0x000000010bc92015 in frame_unwind_register ()
#18 0x000000010bc91d70 in frame_register_unwind ()
#19 0x000000010bc92015 in frame_unwind_register ()
#20 0x000000010bc91d70 in frame_register_unwind ()
#21 0x000000010bc92015 in frame_unwind_register ()
#22 0x000000010bc91d70 in frame_register_unwind ()
#23 0x000000010bc92015 in frame_unwind_register ()
#24 0x000000010bc91d70 in frame_register_unwind ()
#25 0x000000010bc92015 in frame_unwind_register ()
#26 0x000000010bc91d70 in frame_register_unwind ()
#27 0x000000010bc92015 in frame_unwind_register ()
#28 0x000000010bc91d70 in frame_register_unwind ()
#29 0x000000010bc92015 in frame_unwind_register ()
#30 0x000000010bc91d70 in frame_register_unwind ()
#31 0x000000010bc92015 in frame_unwind_register ()
#32 0x000000010bc91d70 in frame_register_unwind ()
#33 0x000000010bc92015 in frame_unwind_register ()
#34 0x000000010bc91d70 in frame_register_unwind ()
#35 0x000000010bc92015 in frame_unwind_register ()
#36 0x000000010bc91d70 in frame_register_unwind ()
#37 0x000000010bc92015 in frame_unwind_register ()
#38 0x000000010bc91d70 in frame_register_unwind ()
#39 0x000000010bc92015 in frame_unwind_register ()
#40 0x000000010bc91d70 in frame_register_unwind ()
#41 0x000000010bc92015 in frame_unwind_register ()
#42 0x000000010bc91d70 in frame_register_unwind ()
#43 0x000000010bc92015 in frame_unwind_register ()

The source code of our modeling application:

void bar(int i)

{

    bar(i+1);

}

 

void foo()

{

    bar(1);

}

 

int main(int argc, const char * argv[])

{

    foo();

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Software Diagnostics Services

Friday, July 13th, 2012

For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:

The First Comprehensive Software Diagnostics Service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 24, Mac OS X)

Saturday, June 9th, 2012

This is a Mac OS X / GDB counterpart to Coincidental Symbolic Information pattern previously described for Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:

(gdb) x $rsp
0x7fff6a162a38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6a162000
[...]
0x7fff6a162960: 0x7fff6a162980 0x7fff6a167922
0x7fff6a162970: 0x0 0x0
0x7fff6a162980: 0x7fff6a162a50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6a162990: 0×1 0×7fff6a162b00
0×7fff6a1629a0: 0×7fff6a162b10 0×7fff6a162bc0
0×7fff6a1629b0: 0×8 0×0
[…]
0×7fff6a162a00: 0×0 0×0
0×7fff6a162a10: 0×0 0×0
0×7fff6a162a20: 0×0 0×0
0×7fff6a162a30: 0×7fff6a162a60 0×7fff8fab9a9c <abort+177>
0×7fff6a162a40: 0×0 0×0
0×7fff6a162a50: 0×7fffffffffdf 0×0
[…]
0×7fff6a163040: 0×35000 0×0
0×7fff6a163050: 0×35000 0×500000007
0×7fff6a163060: 0×7 0×747865745f5f
0×7fff6a163070: 0×0 0×545845545f5f
0×7fff6a163080: 0×0 0×7fff5fc01000 <__dyld_stub_binding_helper>
0×7fff6a163090: 0×22c9d 0xc00001000
0×7fff6a1630a0: 0×0 0×80000400
[…]

(gdb) disass 0×7fff8a31e716
Dump of assembler code for function dyld_stub_binder_:
0×00007fff8a31e709 <dyld_stub_binder_+0>: mov 0×8(%rbp),%rdi
0×00007fff8a31e70d <dyld_stub_binder_+4>: mov 0×10(%rbp),%rsi
0×00007fff8a31e711 <dyld_stub_binder_+8>: callq 0×7fff8a31e86d <_Z21_dyld_fast_stub_entryPvl>
0×00007fff8a31e716 <dyld_stub_binder_+13>: mov %rax,%r11
0×00007fff8a31e719 <dyld_stub_binder_+16>: movdqa 0×40(%rsp),%xmm0
0×00007fff8a31e71f <dyld_stub_binder_+22>: movdqa 0×50(%rsp),%xmm1
0×00007fff8a31e725 <dyld_stub_binder_+28>: movdqa 0×60(%rsp),%xmm2
0×00007fff8a31e72b <dyld_stub_binder_+34>: movdqa 0×70(%rsp),%xmm3
0×00007fff8a31e731 <dyld_stub_binder_+40>: movdqa 0×80(%rsp),%xmm4
0×00007fff8a31e73a <dyld_stub_binder_+49>: movdqa 0×90(%rsp),%xmm5
0×00007fff8a31e743 <dyld_stub_binder_+58>: movdqa 0xa0(%rsp),%xmm6
0×00007fff8a31e74c <dyld_stub_binder_+67>: movdqa 0xb0(%rsp),%xmm7
0×00007fff8a31e755 <dyld_stub_binder_+76>: mov (%rsp),%rdi
0×00007fff8a31e759 <dyld_stub_binder_+80>: mov 0×8(%rsp),%rsi
0×00007fff8a31e75e <dyld_stub_binder_+85>: mov 0×10(%rsp),%rdx
0×00007fff8a31e763 <dyld_stub_binder_+90>: mov 0×18(%rsp),%rcx
0×00007fff8a31e768 <dyld_stub_binder_+95>: mov 0×20(%rsp),%r8
0×00007fff8a31e76d <dyld_stub_binder_+100>: mov 0×28(%rsp),%r9
0×00007fff8a31e772 <dyld_stub_binder_+105>: mov 0×30(%rsp),%rax
0×00007fff8a31e777 <dyld_stub_binder_+110>: add $0xc0,%rsp
0×00007fff8a31e77e <dyld_stub_binder_+117>: pop %rbp
0×00007fff8a31e77f <dyld_stub_binder_+118>: add $0×10,%rsp
0×00007fff8a31e783 <dyld_stub_binder_+122>: jmpq *%r11

(gdb) x/2i 0×7fff8fab9a9c
0×7fff8fab9a9c <abort+177>: mov $0×2710,%edi
0×7fff8fab9aa1 <abort+182>: callq 0×7fff8fab9c43 <usleep$nocancel>

(gdb) disass 0×7fff8fab9a9c-5 0×7fff8fab9a9c
Dump of assembler code from 0×7fff8fab9a97 to 0×7fff8fab9a9c:
0×00007fff8fab9a97 <abort+172>: callq 0×7fff8fb1f54a <dyld_stub_kill>
End of assembler dump.

(gdb) disass 0×7fff5fc01000
Dump of assembler code for function __dyld_stub_binding_helper:
0×00007fff5fc01000 <__dyld_stub_binding_helper+0>: add %al,(%rax)
0×00007fff5fc01002 <__dyld_stub_binding_helper+2>: add %al,(%rax)
0×00007fff5fc01004 <__dyld_stub_binding_helper+4>: add %al,(%rax)
0×00007fff5fc01006 <__dyld_stub_binding_helper+6>: add %al,(%rax)
End of assembler dump.

(gdb) x/10 0×7fff5fc01000-0×10
0×7fff5fc00ff0: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01000 <__dyld_stub_binding_helper>: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01010 <__dyld_offset_to_dyld_all_image_infos>: 0×00000000 0×00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 60, Mac OS X)

Tuesday, June 5th, 2012

This is a Mac OS X / GDB counterpart to Execution Residue pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8616e82a in __kill ()
#1 0x00007fff8fab9a9c in abort ()
#2 0x000000010269dc29 in bar_5 ()
#3 0x000000010269dc39 in bar_4 ()
#4 0x000000010269dc49 in bar_3 ()
#5 0x000000010269dc59 in bar_2 ()
#6 0x000000010269dc69 in bar_1 ()
#7 0x000000010269dc79 in bar ()
#8 0x000000010269dca0 in main (argc=1, argv=0x7fff6229cb00)

(gdb) x $rsp
0x7fff6229ca38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6229c000
0×7fff6229c000: 0×7fff8947b000 0×7fff8947b570
0×7fff6229c010: 0×4f3ee10c 0×7fff90cb0000
0×7fff6229c020: 0×7fff90cb04d0 0×4e938b16
[…]
0×7fff6229c5f0: 0×7fff622d8d80 0×10269d640
0×7fff6229c600: 0×7fff6229cad0 0×7fff622a460b
0×7fff6229c610: 0×100000000 0×269d000
0×7fff6229c620: 0×7fff6229c630 0×10269db59 <foo_8+9>
0×7fff6229c630: 0×7fff6229c640 0×10269db69 <foo_7+9>
0×7fff6229c640: 0×7fff6229c650 0×10269db79 <foo_6+9>
0×7fff6229c650: 0×7fff6229c660 0×10269db89 <foo_5+9>
0×7fff6229c660: 0×7fff6229c670 0×10269db99 <foo_4+9>
0×7fff6229c670: 0×7fff6229c680 0×10269dba9 <foo_3+9>
0×7fff6229c680: 0×7fff6229c690 0×10269dbb9 <foo_2+9>
0×7fff6229c690: 0×7fff6229c6a0 0×10269dbc9 <foo_1+9>
0×7fff6229c6a0: 0×7fff6229cac0 0×10269dbee <foo+30>

0×7fff6229c6b0: 0×0 0×0
0×7fff6229c6c0: 0×0 0×0
0×7fff6229c6d0: 0×0 0×0
0×7fff6229c6e0: 0×0 0×0
[…]
0×7fff6229c8d0: 0×7fff6229c960 0×7fff622b49cd
0×7fff6229c8e0: 0×10269f05c 0×0
0×7fff6229c8f0: 0×7fff622c465c 0×7fff8a31e5c0 <_Z21dyldGlobalLockReleasev>
0×7fff6229c900: 0×7fff8fab99eb <abort> 0×10269f05c
0×7fff6229c910: 0×101000000000000 0×7fff622d2110
0×7fff6229c920: 0×7fff622d8d80 0×10269f078
0×7fff6229c930: 0×7fff622daac8 0×18
0×7fff6229c940: 0×0 0×0
0×7fff6229c950: 0×10269e030 0×0
0×7fff6229c960: 0×7fff6229c980 0×7fff622a1922
0×7fff6229c970: 0×0 0×0
0×7fff6229c980: 0×7fff6229ca50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6229c990: 0×1 0×7fff6229cb00
0×7fff6229c9a0: 0×7fff6229cb10 0xe223ea612ddc10b7
0×7fff6229c9b0: 0×8 0×0
0×7fff6229c9c0: 0xe223ea612ddc10b7 0×0
0×7fff6229c9d0: 0×0 0×0
0×7fff6229c9e0: 0×585f5f00474e414c 0×20435058005f4350
0×7fff6229c9f0: 0×0 0×0
0×7fff6229ca00: 0×0 0×0
0×7fff6229ca10: 0×0 0×0
0×7fff6229ca20: 0×0 0×0
0×7fff6229ca30: 0×7fff6229ca60 0×7fff8fab9a9c <abort+177>
0×7fff6229ca40: 0×0 0×0
0×7fff6229ca50: 0×7fffffffffdf 0×0
0×7fff6229ca60: 0×7fff6229ca70 0×10269dc29 <bar_5+9>
0×7fff6229ca70: 0×7fff6229ca80 0×10269dc39 <bar_4+9>
0×7fff6229ca80: 0×7fff6229ca90 0×10269dc49 <bar_3+9>
0×7fff6229ca90: 0×7fff6229caa0 0×10269dc59 <bar_2+9>
0×7fff6229caa0: 0×7fff6229cab0 0×10269dc69 <bar_1+9>
0×7fff6229cab0: 0×7fff6229cac0 0×10269dc79 <bar+9>
0×7fff6229cac0: 0×7fff6229cae0 0×10269dca0 <main+32>
0×7fff6229cad0: 0×7fff6229cb00 0×1
0×7fff6229cae0: 0×7fff6229caf0 0×10269db34 <start+52>

0×7fff6229caf0: 0×0 0×1
0×7fff6229cb00: 0×7fff6229cc48 0×0
0×7fff6229cb10: 0×7fff6229ccae 0×7fff6229ccca
[…]

Here’s the source code of the modeling application:

#define def_call(name,x,y) void name##_##x() { name##_##y(); }

#define def_final(name,x) void name##_##x() { }

#define def_final_abort(name,x) void name##_##x() { abort(); }

#define def_init(name,y) void name() { name##_##y(); }

#define def_init_alloc(name,y,size) void name() { int arr[size]; name##_##y(); *arr=0; }

def_final(foo,9)

def_call(foo,8,9)

def_call(foo,7,8)

def_call(foo,6,7)

def_call(foo,5,6)

def_call(foo,4,5)

def_call(foo,3,4)

def_call(foo,2,3)

def_call(foo,1,2)

def_init_alloc(foo,1,256)

def_final_abort(bar,5)

def_call(bar,4,5)

def_call(bar,3,4)

def_call(bar,2,3)

def_call(bar,1,2)

def_init(bar,1)

int main(int argc, const char * argv[])

{

    foo();

    bar();

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 23a, Mac OS X)

Tuesday, May 29th, 2012

This is a Mac OS X / GDB counterpart to Double Free (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e13f84c in free ()
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)

(gdb) x/2i 0x00000001035a8ef4-8
0x1035a8eec : mov -0×20(%rbp),%edi
0×1035a8eef : callq 0×1035a8f06

(gdb) frame 3
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)
at .../DoubleFree/main.c:23
23 free(p2);
Current language: auto; currently minimal

(gdb) x/g $rbp-0x20
0x7fff631a7ae0: 0x00007fe6a8801400

(gdb) x/2w 0x00007fe6a8801400
0x7fe6a8801400: 0x00000000 0xb0000000

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    free(p2);

    free(p1);

    free(p2);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 2, Mac OS X)

Sunday, May 27th, 2012

This is a Mac OS X / GDB counterpart to Dynamic Memory Corruption (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e1024ac in szone_error ()
#3 0x00007fff8e1024e8 in free_list_checksum_botch ()
#4 0x00007fff8e102a7b in small_free_list_remove_ptr ()
#5 0x00007fff8e106bf7 in szone_free_definite_size ()
#6 0x00007fff8e13f789 in free ()
#7 0x000000010afafe23 in main (argc=1, argv=0x7fff6abaeb08)

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    char *p3 = (char *) malloc (1024);

    printf(“p3 = %p\n”, p3);

 

    char *p4 = (char *) malloc (1024);

    printf(“p4 = %p\n”, p4);

 

    char *p5 = (char *) malloc (1024);

    printf(“p5 = %p\n”, p5);

 

    char *p6 = (char *) malloc (1024);

    printf(“p6 = %p\n”, p6);

 

    char *p7 = (char *) malloc (1024);

    printf(“p7 = %p\n”, p7);

 

    free(p6);

    free(p4);

    free(p2);

 

    printf(“Hello Crash!\n”);        

    strcpy(p2, “Hello Crash!”);

    strcpy(p4, “Hello Crash!”);

    strcpy(p6, “Hello Crash!”);

 

    p2 = (char *) malloc (512);

    printf(“p2 = %p\n”, p2);

 

    p4 = (char *) malloc (1024);

    printf(“p4 = %p\n”, p4);

 

    p6 = (char *) malloc (512);

    printf(“p6 = %p\n”, p6);

 

    free (p7);

    free (p6);

    free (p5);

    free (p4);

    free (p3);

    free (p2);

    free (p1);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 14, Mac OS X)

Wednesday, May 9th, 2012

This is a Mac OS X / GDB counterpart to Spiking Thread pattern previously described for Windows platforms:

(gdb) info threads
4 0×00007fff85b542df in sqrt$fenv_access_off ()
3 0×00007fff8616ee42 in __semwait_signal ()
2 0×00007fff8616ee42 in __semwait_signal ()
* 1 0×00007fff8616ee42 in __semwait_signal ()

We notice a non-waiting thread and switch to it:

(gdb) thread 4
[Switching to thread 4 (core thread 3)]
0x00007fff85b542df in sqrt$fenv_access_off ()

(gdb) bt
#0  0x00007fff85b542df in sqrt$fenv_access_off ()
#1  0×000000010cc85dc9 in thread_three (arg=0×7fff6c884ac0)
#2  0×00007fff8fac68bf in _pthread_start ()
#3  0×00007fff8fac9b75 in thread_start ()

If we disassemble the return address for thread_three function to come back from sqrt call we see an infinite loop:

(gdb) disass 0x000000010cc85dc9
Dump of assembler code for function thread_three:
0x000000010cc85db0 <thread_three+0>: push   %rbp
0×000000010cc85db1 <thread_three+1>: mov    %rsp,%rbp
0×000000010cc85db4 <thread_three+4>: sub    $0×10,%rsp
0×000000010cc85db8 <thread_three+8>: mov    %rdi,-0×10(%rbp)
0×000000010cc85dbc <thread_three+12>: mov    -0×10(%rbp),%ax
0×000000010cc85dc0 <thread_three+16>: movsd  (%rax),%xmm0
0×000000010cc85dc4 <thread_three+20>: callq  0×10cc85eac <dyld_stub_sqrt>
0×000000010cc85dc9 <thread_three+25>: mov    -0×10(%rbp),%rax
0×000000010cc85dcd <thread_three+29>: movsd  %xmm0,(%rax)
0×000000010cc85dd1 <thread_three+33>: jmpq   0×10cc85dbc <thread_three+12>
End of assembler dump.

Here’s the source code of the modeling application:

void * thread_one (void *arg)

{

    while (1)

    {

       sleep (1);

    }

 

    return 0;

}

 

void * thread_two (void *arg)

{

    while (1)

    {

        sleep (2);

    }

 

    return 0;

}

 

void * thread_three (void *arg)

{

    while (1)

    {

        *(double*)arg=sqrt(*(double *)arg);

    }

 

    return 0;

}

 

int main(int argc, const char * argv[])

{

    pthread_t threadID_one, threadID_two, threadID_three;

 

    double result = 0xffffffff;

 

    pthread_create (&threadID_one, NULL, thread_one, NULL);

    pthread_create (&threadID_two, NULL, thread_two, NULL);

    pthread_create (&threadID_three, NULL, thread_three,

       &result);

 

    pthread_join(threadID_three, NULL);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 6a, Mac OS X)

Thursday, May 3rd, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (code) pattern previously described for Windows platforms:

(gdb) bt
#0 0×0000000000000000 in ?? ()
#1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) disass 0×000000010e8cce73-3 0×000000010e8cce73
Dump of assembler code from 0×10e8cce70 to 0×10e8cce73:
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

(gdb) info r rdi
rdi 0x7fff6e4cbac0 140735043910336

(gdb) x/2 0x7fff6e4cbac0
0x7fff6e4cbac0: 0x0000000a 0×00000000

(gdb) p/x *($rdi+8)
$7 = 0×0

(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) ptype MYSTRUCT
type = struct _MyStruct_tag {
int data;
PFUNC pfunc;
}

(gdb) print {MYSTRUCT}0×7fff6e4cbac0
$2 = {data = 10, pfunc = 0}

Here’s the source code of the modeling application:

typedef void (*PFUNC)(void);

 

typedef struct _MyStruct_tag

{

    int   data;

    PFUNC pfunc;

} MYSTRUCT;

 

void bar(MYSTRUCT *ps)

{

    ps->pfunc();

}

 

void foo(MYSTRUCT *ps)

{

    bar(ps);

}

 

int main(int argc, const char * argv[])

{

    MYSTRUCT pstruct = {10, NULL};

 

    foo(&pstruct);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 6b, Mac OS X)

Sunday, March 25th, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (data) pattern previously described for Windows platforms:

(gdb) bt
#0  0×000000010d3b0e90 in bar () at main.c:15
#1  0×000000010d3b0ea9 in foo () at main.c:20
#2  0×000000010d3b0ec4 in main (argc=1,
argv=0×7fff6cfafbf8) at main.c:25

(gdb) disassemble
Dump of assembler code for function bar:
0x000000010d3b0e80 <bar+0>: push   %rbp
0×000000010d3b0e81 <bar+1>: mov    %rsp,%rbp
0×000000010d3b0e84 <bar+4>: movq   $0×0,-0×8(%rbp)
0×000000010d3b0e8c <bar+12>: mov    -0×8(%rbp),%rax
0×000000010d3b0e90 <bar+16>: movl   $0×1,(%rax)
0×000000010d3b0e96 <bar+22>: pop    %bp
0×000000010d3b0e97 <bar+23>: retq
End of assembler dump.

(gdb) p/x $rax
$1 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis