Archive for the ‘GDB for WinDbg Users’ Category

Crash Dump Analysis Patterns (Part 180, Linux)

Monday, December 14th, 2015

This is Linux variant of Paratext pattern for Mac OS X. Because of debugger tool limitations additional software logs and the output of other tools may help in memory dump analysis. Typical examples of such pattern usage can be the list of modules with version and path info, application crash specific information from instrumentation tools such as valgrind, memory region names with attribution and boundaries, and CPU usage information. For example, top and pmap commands output:

14039: ./App1.shared
0000000000400000 4K r-x-- /home/training/ALCDA/App1/App1.shared
0000000000600000 4K rw--- /home/training/ALCDA/App1/App1.shared
0000000000611000 132K rw--- [ anon ]
00007fe8999a6000 4K ----- [ anon ]
00007fe8999a7000 8192K rw--- [ anon ]
00007fe89a1a7000 4K ----- [ anon ]
00007fe89a1a8000 8192K rw--- [ anon ]
00007fe89a9a8000 4K ----- [ anon ]
00007fe89a9a9000 8192K rw--- [ anon ]
00007fe89b1a9000 4K ----- [ anon ]
00007fe89b1aa000 8192K rw--- [ anon ]
00007fe89b9aa000 4K ----- [ anon ]
00007fe89b9ab000 8192K rw--- [ anon ]
00007fe89c1ab000 1540K r-x-- /lib/x86_64-linux-gnu/libc-2.13.so
00007fe89c32c000 2048K ----- /lib/x86_64-linux-gnu/libc-2.13.so
00007fe89c52c000 16K r---- /lib/x86_64-linux-gnu/libc-2.13.so
00007fe89c530000 4K rw--- /lib/x86_64-linux-gnu/libc-2.13.so
00007fe89c531000 20K rw--- [ anon ]
00007fe89c536000 92K r-x-- /lib/x86_64-linux-gnu/libpthread-2.13.so
00007fe89c54d000 2044K ----- /lib/x86_64-linux-gnu/libpthread-2.13.so
00007fe89c74c000 4K r---- /lib/x86_64-linux-gnu/libpthread-2.13.so
00007fe89c74d000 4K rw--- /lib/x86_64-linux-gnu/libpthread-2.13.so
00007fe89c74e000 16K rw--- [ anon ]
00007fe89c752000 128K r-x-- /lib/x86_64-linux-gnu/ld-2.13.so
00007fe89c966000 12K rw--- [ anon ]
00007fe89c96f000 8K rw--- [ anon ]
00007fe89c971000 4K r---- /lib/x86_64-linux-gnu/ld-2.13.so
00007fe89c972000 4K rw--- /lib/x86_64-linux-gnu/ld-2.13.so
00007fe89c973000 4K rw--- [ anon ]
00007ffd458c1000 132K rw--- [ stack ]
00007ffd459e9000 4K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 47208K

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 180, Mac OS X)

Saturday, July 28th, 2012

This is the first pattern that emerged after applying the same pattern-driven software diagnostics methodology to Mac OS X. I had problems using GDB which is so portable that hardly has operating system support like WinDbg has. Fortunately, I found a workaround by complementing core dumps with logs and reports from OS such as crash reports and vmmap data. I call this pattern Paratext which I borrowed from the concept of an extended software trace and software narratology where it borrowed the same concept from literary interpretation (paratext). Typical examples of such pattern usage can be the list of modules with version and path info, application crash specific information, memory region names with attribution and boundaries:

// from .crash reports

0x108f99000 - 0x109044ff7 com.apple.FontBook (198.4 - 198) <7244D36E-4563-3E42-BA46-1F279D30A6CE> /Applications/Font Book.app/Contents/MacOS/Font Book

Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000

Application Specific Information:
objc[195]: garbage collection is OFF
*** error for object 0x7fd7fb818e08: incorrect checksum for freed object - object was probably modified after being freed.

// from vmmap logs

[...]
==== Writable regions for process 966
[...]
Stack 0000000101f71000-0000000101ff3000 [ 520K] rw-/rwx SM=PRV thread 1
MALLOC_LARGE 0000000103998000-00000001039b8000 [ 128K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00000001039b9000-00000001039bb000 [ 8K] rw-/rwx SM=PRV
mapped file 0000000103a05000-0000000103f32000 [ 5300K] rw-/rwx SM=COW ...box.framework/Versions/A/Resources/Extras2.rsrc
mapped file 0000000104409000-00000001046d2000 [ 2852K] rw-/rwx SM=COW /System/Library/Fonts/Helvetica.dfont
MALLOC_LARGE 0000000104f6e000-0000000104f8e000 [ 128K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_LARGE (freed) 0000000108413000-0000000108540000 [ 1204K] rw-/rwx SM=COW
MALLOC_LARGE (freed) 0000000108540000-0000000108541000 [ 4K] rw-/rwx SM=PRV
MALLOC_TINY 00007fefe0c00000-00007fefe0d00000 [ 1024K] rw-/rwx SM=COW DefaultMallocZone_0x101e6e000
MALLOC_TINY 00007fefe0d00000-00007fefe0e00000 [ 1024K] rw-/rwx SM=PRV DispatchContinuations_0x101f38000
MALLOC_TINY 00007fefe0e00000-00007fefe0f00000 [ 1024K] rw-/rwx SM=COW DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1000000-00007fefe107b000 [ 492K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe107b000-00007fefe1083000 [ 32K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1083000-00007fefe1149000 [ 792K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe1149000-00007fefe1166000 [ 116K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe1166000-00007fefe1800000 [ 6760K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1800000-00007fefe18ff000 [ 1020K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_SMALL (freed) 00007fefe18ff000-00007fefe1901000 [ 8K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
MALLOC_SMALL 00007fefe1901000-00007fefe2000000 [ 7164K] rw-/rwx SM=ZER DefaultMallocZone_0x101e6e000
MALLOC_TINY (freed) 00007fefe2000000-00007fefe2100000 [ 1024K] rw-/rwx SM=PRV DispatchContinuations_0x101f38000
MALLOC_TINY 00007fefe2100000-00007fefe2200000 [ 1024K] rw-/rwx SM=PRV DefaultMallocZone_0x101e6e000
Stack 00007fff61186000-00007fff61985000 [ 8188K] rw-/rwx SM=ZER thread 0
Stack 00007fff61985000-00007fff61986000 [ 4K] rw-/rwx SM=COW
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 18, Mac OS X)

Friday, July 20th, 2012

This is a Mac OS X / GDB counterpart to Truncated Dump pattern previously described for Windows platforms:

(gdb) info threads
Cannot access memory at address 0x7fff885e9e42
4 0x00007fff885e9e42 in ?? ()
3 0x00007fff885e9e42 in ?? ()
2 0x00007fff885e9e42 in ?? ()
* 1 0x00007fff885e9e42 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0x00007fff885e9e42 in ?? ()

(gdb) disass 0x00007fff885e9e42
No function contains specified address.

(gdb) info r rsp
rsp 0x7fff67fe8a18 0x7fff67fe8a18

(gdb) x/100a 0x7fff67fe8a18
0x7fff67fe8a18: Cannot access memory at address 0x7fff67fe8a18

This often happens if there is no space to save a full core dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 77, Mac OS X)

Friday, July 20th, 2012

This is a Mac OS X / GDB counterpart to C++ Exception pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff88bd582a in __kill ()
#1 0x00007fff8c184a9c in abort ()
#2 0x00007fff852f57bc in abort_message ()
#3 0x00007fff852f2fcf in default_terminate ()
#4 0x00007fff852f3001 in safe_handler_caller ()
#5 0x00007fff852f305c in std::terminate ()
#6 0×00007fff852f4152 in __cxa_throw ()
#7 0×000000010e402be8 in bar ()
#8 0×000000010e402c99 in foo ()
#9 0×000000010e402cbb in main (argc=1, argv=0×7fff6e001b18)

The modeling application source code:

class Exception

{

    int code;

    std::string description;

 

public:

    Exception(int _code, std::string _desc) : code(_code), description(_desc) {}

};

 

void bar()

{

    throw new Exception(5, “Access Denied”);

}

 

void foo()

{

    bar();

}

 

int main(int argc, const char * argv[])

{

    foo();

    return 0;

}  

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 36, Mac OS X)

Thursday, July 19th, 2012

This is a Mac OS X / GDB counterpart to Local Buffer Overflow pattern previously described for Windows platforms. Most of the time simple mistakes in using memory and string manipulation functions are easily detected by runtime:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff8325a89f in __chk_fail ()
#3 0×00007fff8325a83e in __memcpy_chk ()

#4 0×000000010914edf3 in bar ()
#5 0×000000010914ee5e in foo ()
#6 0×000000010914ee9b in main (argc=1, argv=0×7fff68d4daf0)

This detection happens in a default optimized release version as well:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff8325a89f in __chk_fail ()
#3 0×00007fff8325a83e in __memcpy_chk ()

#4 0×000000010f59cea8 in bar [inlined] ()
#5 0×000000010f59cea8 in foo [inlined] ()
#6 0×000000010f59cea8 in main (argc=,
argv=)

The more sophisticated example which overwrites stack trace without being detected involves overwriting indirectly via a pointer to a local buffer passed to the called function. In such cases we might see incorrect and truncated stack traces:

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff83285070 in __stack_chk_fail ()
#3 0×000000010524de77 in foo ()
#4 0xca4000007fff64e5 in ?? ()

(gdb) bt
#0 0x00007fff885e982a in __kill ()
#1 0x00007fff83288b6c in __abort ()
#2 0×00007fff83285070 in __stack_chk_fail ()
#3 0×0000000105ad8df7 in foo ()

Inspection of the raw stack shows ASCII-like memory values around foo symbolic reference instead of expected main and start functions:

(gdb) info r rsp
rsp 0x7fff656d79d8 0x7fff656d79d8

(gdb) x/100a 0x7fff656d79d8
0x7fff656d79d8: 0x7fff83288b6c <__abort+193> 0x0
0x7fff656d79e8: 0x0 0xffffffdf
0x7fff656d79f8: 0x7fff656d7a40 0x7fff656d7a80
0x7fff656d7a08: 0x7fff83285070 <__guard_setup> 0x6675426c61636f4c
0x7fff656d7a18: 0x7265764f726566 0x0
0x7fff656d7a28: 0x0 0x0
0x7fff656d7a38: 0x0 0x73205d343336325b
0x7fff656d7a48: 0x65766f206b636174 0x776f6c6672
0x7fff656d7a58: 0x0 0x0
0x7fff656d7a68: 0x0 0x343336326d7ab0
0x7fff656d7a78: 0x0 0x7fff656d7ab0
0x7fff656d7a88: 0x105ad8df7 0xb1887b8452358ac4
0×7fff656d7a98: 0×794d000000000000 0×6769422077654e20
0×7fff656d7aa8: 0×6666754220726567 0×7265
0×7fff656d7ab8: 0×0 0×0
0×7fff656d7ac8: 0×0 0×0
0×7fff656d7ad8: 0×0 0×0
0×7fff656d7ae8: 0×0 0×0
[…]

The modeling application source code:

void bar(char *buffer)

{

      char data[100] = “My New Bigger Buffer”

      memcpy (buffer, data, sizeof(data));

}

 

void foo()

{

    char data[10] = “My Buffer”;

    bar(data);

}

 

int main(int argc, const char * argv[])

{

    foo();

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 78a, Mac OS X)

Wednesday, July 18th, 2012

This is a Mac OS X / GDB counterpart to Divide by Zero (user mode) pattern previously described for Windows platforms:

(gdb) bt
#0 0×000000010d3ebe9e in bar (a=1, b=0)
#1 0×000000010d3ebec3 in foo ()
#2 0×000000010d3ebeeb in main (argc=1, argv=0×7fff6cfeab18)

(gdb) x/i 0×000000010d3ebe9e
0×10d3ebe9e : idiv %esi

(gdb) info r rsi
rsi 0×0 0

The modeling application source code:

int bar(int a, int b)

{

    return a/b;

}

 

int foo()

{

    return bar(1,0);

}

 

int main(int argc, const char * argv[])

{

    return foo();

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 16b, Mac OS X)

Tuesday, July 17th, 2012

This is a Mac OS X / GDB counterpart to Stack Overflow (user mode) pattern previously described for Windows platforms:

(gdb) bt 10
#0 0x0000000105dafea8 in bar (i=0)
#1 0x0000000105dafeb9 in bar (i=262102)
#2 0x0000000105dafeb9 in bar (i=262101)
#3 0x0000000105dafeb9 in bar (i=262100)
#4 0x0000000105dafeb9 in bar (i=262099)
#5 0x0000000105dafeb9 in bar (i=262098)
#6 0x0000000105dafeb9 in bar (i=262097)
#7 0x0000000105dafeb9 in bar (i=262096)
#8 0x0000000105dafeb9 in bar (i=262095)
#9 0x0000000105dafeb9 in bar (i=262094)
(More stack frames follow...)

There are at least 262,102 frames so we don’t attempt to list them all. What we’d like to do is to get stack trace boundaries from the list of sections based on the current stack pointer address and dump the upper part of it (the stack grows from higher addresses to the lower ones) to get bottom initial stack traces:

(gdb) x $rsp
0×7fff651aeff0: 0×00000000

Because this is a stack overflow we expect RSP went out of page bounds so we expect the lowest address being 0×7fff651af000.

(gdb) maint info sections
[...]
Core file:
`/cores/core.2763', file type mach-o-le.
[...]
0x0000000105e00000->0x0000000105f00000 at 0x00035000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff619af000->0x00007fff651af000 at 0x00135000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff651af000->0×00007fff659af000 at 0×03935000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff659af000->0×00007fff659e4000 at 0×04135000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×00007fff659e4000->0×00007fff659e6000 at 0×0416a000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
[…]

(gdb) x/250a 0×00007fff659af000-2000
0×7fff659ae830: 0×0 0×1500000000
0×7fff659ae840: 0×7fff659ae860 0×105dafeb9 <bar+25>
0×7fff659ae850: 0×0 0×1400000000
0×7fff659ae860: 0×7fff659ae880 0×105dafeb9 <bar+25>
0×7fff659ae870: 0×0 0×1300000000
0×7fff659ae880: 0×7fff659ae8a0 0×105dafeb9 <bar+25>
0×7fff659ae890: 0×0 0×1200000000
0×7fff659ae8a0: 0×7fff659ae8c0 0×105dafeb9 <bar+25>
0×7fff659ae8b0: 0×0 0×1100000000
0×7fff659ae8c0: 0×7fff659ae8e0 0×105dafeb9 <bar+25>
0×7fff659ae8d0: 0×0 0×1000000000
0×7fff659ae8e0: 0×7fff659ae900 0×105dafeb9 <bar+25>
0×7fff659ae8f0: 0×0 0xf00000000
0×7fff659ae900: 0×7fff659ae920 0×105dafeb9 <bar+25>
0×7fff659ae910: 0×0 0xe00000000
0×7fff659ae920: 0×7fff659ae940 0×105dafeb9 <bar+25>
0×7fff659ae930: 0×0 0xd00000000
0×7fff659ae940: 0×7fff659ae960 0×105dafeb9 <bar+25>
0×7fff659ae950: 0×0 0xc00000000
0×7fff659ae960: 0×7fff659ae980 0×105dafeb9 <bar+25>
0×7fff659ae970: 0×0 0xb00000000
0×7fff659ae980: 0×7fff659ae9a0 0×105dafeb9 <bar+25>
0×7fff659ae990: 0×0 0xa00000000
0×7fff659ae9a0: 0×7fff659ae9c0 0×105dafeb9 <bar+25>
0×7fff659ae9b0: 0×0 0×900000000
0×7fff659ae9c0: 0×7fff659ae9e0 0×105dafeb9 <bar+25>
0×7fff659ae9d0: 0×0 0×800000000
0×7fff659ae9e0: 0×7fff659aea00 0×105dafeb9 <bar+25>
0×7fff659ae9f0: 0×0 0×700000000
0×7fff659aea00: 0×7fff659aea20 0×105dafeb9 <bar+25>
0×7fff659aea10: 0×0 0×600000000
0×7fff659aea20: 0×7fff659aea40 0×105dafeb9 <bar+25>
0×7fff659aea30: 0×0 0×5659b9fe0
0×7fff659aea40: 0×7fff659aea60 0×105dafeb9 <bar+25
0×7fff659aea50: 0×7fff659aea70 0×4659bd31f
0×7fff659aea60: 0×7fff659aea80 0×105dafeb9 <bar+25>
0×7fff659aea70: 0×7fff659aeaf0 0×3659b031a
0×7fff659aea80: 0×7fff659aeaa0 0×105dafeb9 <bar+25>
0×7fff659aea90: 0×7fff659af5c0 0×200000000
0×7fff659aeaa0: 0×7fff659aeac0 0×105dafeb9 <bar+25>
0×7fff659aeab0: 0×100000000 0×1659aeb18
0×7fff659aeac0: 0×7fff659aead0 0×105dafece <foo+14>
0×7fff659aead0: 0×7fff659aeaf0 0×105dafeeb <main+27>
0×7fff659aeae0: 0×7fff659aeb18 0×1
—Type to continue, or q to quit—
0×7fff659aeaf0: 0×7fff659aeb08 0×105dafe94 <start+52>
0×7fff659aeb00: 0×0 0×0
[…]
0×7fff659aeff0: 0×3139336561303363 0×316235

Interesting if we set the lowest frame down and try to get register info GDB core dumps:

(gdb) frame 262102
#262102 0x0000000105dafeb9 in bar (i=1)
13 bar(i+1);
(gdb) info r
Segmentation fault: 11 (core dumped)

Looking its core dump show that it also experienced stack overflow:

(gdb) bt
#0 0x00007fff8c1bacf0 in __sfvwrite ()
#1 0x00007fff8c189947 in __vfprintf ()
#2 0x00007fff8c184edb in vsnprintf_l ()
#3 0x00007fff8c1566be in __sprintf_chk ()
#4 0x000000010bd14d15 in print_displacement ()
#5 0x000000010bd10ddf in OP_E ()
#6 0x000000010bd13f9b in print_insn ()
#7 0x000000010bc164ce in length_of_this_instruction ()
#8 0x000000010bc9e296 in x86_analyze_prologue ()
#9 0x000000010bc9f1f3 in x86_frame_prev_register ()
#10 0x000000010bc91d70 in frame_register_unwind ()
#11 0x000000010bc92015 in frame_unwind_register ()
#12 0x000000010bc91d70 in frame_register_unwind ()
#13 0x000000010bc92015 in frame_unwind_register ()
#14 0x000000010bc91d70 in frame_register_unwind ()
#15 0x000000010bc92015 in frame_unwind_register ()
#16 0x000000010bc91d70 in frame_register_unwind ()
#17 0x000000010bc92015 in frame_unwind_register ()
#18 0x000000010bc91d70 in frame_register_unwind ()
#19 0x000000010bc92015 in frame_unwind_register ()
#20 0x000000010bc91d70 in frame_register_unwind ()
#21 0x000000010bc92015 in frame_unwind_register ()
#22 0x000000010bc91d70 in frame_register_unwind ()
#23 0x000000010bc92015 in frame_unwind_register ()
#24 0x000000010bc91d70 in frame_register_unwind ()
#25 0x000000010bc92015 in frame_unwind_register ()
#26 0x000000010bc91d70 in frame_register_unwind ()
#27 0x000000010bc92015 in frame_unwind_register ()
#28 0x000000010bc91d70 in frame_register_unwind ()
#29 0x000000010bc92015 in frame_unwind_register ()
#30 0x000000010bc91d70 in frame_register_unwind ()
#31 0x000000010bc92015 in frame_unwind_register ()
#32 0x000000010bc91d70 in frame_register_unwind ()
#33 0x000000010bc92015 in frame_unwind_register ()
#34 0x000000010bc91d70 in frame_register_unwind ()
#35 0x000000010bc92015 in frame_unwind_register ()
#36 0x000000010bc91d70 in frame_register_unwind ()
#37 0x000000010bc92015 in frame_unwind_register ()
#38 0x000000010bc91d70 in frame_register_unwind ()
#39 0x000000010bc92015 in frame_unwind_register ()
#40 0x000000010bc91d70 in frame_register_unwind ()
#41 0x000000010bc92015 in frame_unwind_register ()
#42 0x000000010bc91d70 in frame_register_unwind ()
#43 0x000000010bc92015 in frame_unwind_register ()

The source code of our modeling application:

void bar(int i)

{

    bar(i+1);

}

 

void foo()

{

    bar(1);

}

 

int main(int argc, const char * argv[])

{

    foo();

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 24, Mac OS X)

Saturday, June 9th, 2012

This is a Mac OS X / GDB counterpart to Coincidental Symbolic Information pattern previously described for Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:

(gdb) x $rsp
0x7fff6a162a38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6a162000
[...]
0x7fff6a162960: 0x7fff6a162980 0x7fff6a167922
0x7fff6a162970: 0x0 0x0
0x7fff6a162980: 0x7fff6a162a50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6a162990: 0×1 0×7fff6a162b00
0×7fff6a1629a0: 0×7fff6a162b10 0×7fff6a162bc0
0×7fff6a1629b0: 0×8 0×0
[…]
0×7fff6a162a00: 0×0 0×0
0×7fff6a162a10: 0×0 0×0
0×7fff6a162a20: 0×0 0×0
0×7fff6a162a30: 0×7fff6a162a60 0×7fff8fab9a9c <abort+177>
0×7fff6a162a40: 0×0 0×0
0×7fff6a162a50: 0×7fffffffffdf 0×0
[…]
0×7fff6a163040: 0×35000 0×0
0×7fff6a163050: 0×35000 0×500000007
0×7fff6a163060: 0×7 0×747865745f5f
0×7fff6a163070: 0×0 0×545845545f5f
0×7fff6a163080: 0×0 0×7fff5fc01000 <__dyld_stub_binding_helper>
0×7fff6a163090: 0×22c9d 0xc00001000
0×7fff6a1630a0: 0×0 0×80000400
[…]

(gdb) disass 0×7fff8a31e716
Dump of assembler code for function dyld_stub_binder_:
0×00007fff8a31e709 <dyld_stub_binder_+0>: mov 0×8(%rbp),%rdi
0×00007fff8a31e70d <dyld_stub_binder_+4>: mov 0×10(%rbp),%rsi
0×00007fff8a31e711 <dyld_stub_binder_+8>: callq 0×7fff8a31e86d <_Z21_dyld_fast_stub_entryPvl>
0×00007fff8a31e716 <dyld_stub_binder_+13>: mov %rax,%r11
0×00007fff8a31e719 <dyld_stub_binder_+16>: movdqa 0×40(%rsp),%xmm0
0×00007fff8a31e71f <dyld_stub_binder_+22>: movdqa 0×50(%rsp),%xmm1
0×00007fff8a31e725 <dyld_stub_binder_+28>: movdqa 0×60(%rsp),%xmm2
0×00007fff8a31e72b <dyld_stub_binder_+34>: movdqa 0×70(%rsp),%xmm3
0×00007fff8a31e731 <dyld_stub_binder_+40>: movdqa 0×80(%rsp),%xmm4
0×00007fff8a31e73a <dyld_stub_binder_+49>: movdqa 0×90(%rsp),%xmm5
0×00007fff8a31e743 <dyld_stub_binder_+58>: movdqa 0xa0(%rsp),%xmm6
0×00007fff8a31e74c <dyld_stub_binder_+67>: movdqa 0xb0(%rsp),%xmm7
0×00007fff8a31e755 <dyld_stub_binder_+76>: mov (%rsp),%rdi
0×00007fff8a31e759 <dyld_stub_binder_+80>: mov 0×8(%rsp),%rsi
0×00007fff8a31e75e <dyld_stub_binder_+85>: mov 0×10(%rsp),%rdx
0×00007fff8a31e763 <dyld_stub_binder_+90>: mov 0×18(%rsp),%rcx
0×00007fff8a31e768 <dyld_stub_binder_+95>: mov 0×20(%rsp),%r8
0×00007fff8a31e76d <dyld_stub_binder_+100>: mov 0×28(%rsp),%r9
0×00007fff8a31e772 <dyld_stub_binder_+105>: mov 0×30(%rsp),%rax
0×00007fff8a31e777 <dyld_stub_binder_+110>: add $0xc0,%rsp
0×00007fff8a31e77e <dyld_stub_binder_+117>: pop %rbp
0×00007fff8a31e77f <dyld_stub_binder_+118>: add $0×10,%rsp
0×00007fff8a31e783 <dyld_stub_binder_+122>: jmpq *%r11

(gdb) x/2i 0×7fff8fab9a9c
0×7fff8fab9a9c <abort+177>: mov $0×2710,%edi
0×7fff8fab9aa1 <abort+182>: callq 0×7fff8fab9c43 <usleep$nocancel>

(gdb) disass 0×7fff8fab9a9c-5 0×7fff8fab9a9c
Dump of assembler code from 0×7fff8fab9a97 to 0×7fff8fab9a9c:
0×00007fff8fab9a97 <abort+172>: callq 0×7fff8fb1f54a <dyld_stub_kill>
End of assembler dump.

(gdb) disass 0×7fff5fc01000
Dump of assembler code for function __dyld_stub_binding_helper:
0×00007fff5fc01000 <__dyld_stub_binding_helper+0>: add %al,(%rax)
0×00007fff5fc01002 <__dyld_stub_binding_helper+2>: add %al,(%rax)
0×00007fff5fc01004 <__dyld_stub_binding_helper+4>: add %al,(%rax)
0×00007fff5fc01006 <__dyld_stub_binding_helper+6>: add %al,(%rax)
End of assembler dump.

(gdb) x/10 0×7fff5fc01000-0×10
0×7fff5fc00ff0: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01000 <__dyld_stub_binding_helper>: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01010 <__dyld_offset_to_dyld_all_image_infos>: 0×00000000 0×00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 60, Mac OS X)

Tuesday, June 5th, 2012

This is a Mac OS X / GDB counterpart to Execution Residue pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8616e82a in __kill ()
#1 0x00007fff8fab9a9c in abort ()
#2 0x000000010269dc29 in bar_5 ()
#3 0x000000010269dc39 in bar_4 ()
#4 0x000000010269dc49 in bar_3 ()
#5 0x000000010269dc59 in bar_2 ()
#6 0x000000010269dc69 in bar_1 ()
#7 0x000000010269dc79 in bar ()
#8 0x000000010269dca0 in main (argc=1, argv=0x7fff6229cb00)

(gdb) x $rsp
0x7fff6229ca38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6229c000
0×7fff6229c000: 0×7fff8947b000 0×7fff8947b570
0×7fff6229c010: 0×4f3ee10c 0×7fff90cb0000
0×7fff6229c020: 0×7fff90cb04d0 0×4e938b16
[…]
0×7fff6229c5f0: 0×7fff622d8d80 0×10269d640
0×7fff6229c600: 0×7fff6229cad0 0×7fff622a460b
0×7fff6229c610: 0×100000000 0×269d000
0×7fff6229c620: 0×7fff6229c630 0×10269db59 <foo_8+9>
0×7fff6229c630: 0×7fff6229c640 0×10269db69 <foo_7+9>
0×7fff6229c640: 0×7fff6229c650 0×10269db79 <foo_6+9>
0×7fff6229c650: 0×7fff6229c660 0×10269db89 <foo_5+9>
0×7fff6229c660: 0×7fff6229c670 0×10269db99 <foo_4+9>
0×7fff6229c670: 0×7fff6229c680 0×10269dba9 <foo_3+9>
0×7fff6229c680: 0×7fff6229c690 0×10269dbb9 <foo_2+9>
0×7fff6229c690: 0×7fff6229c6a0 0×10269dbc9 <foo_1+9>
0×7fff6229c6a0: 0×7fff6229cac0 0×10269dbee <foo+30>

0×7fff6229c6b0: 0×0 0×0
0×7fff6229c6c0: 0×0 0×0
0×7fff6229c6d0: 0×0 0×0
0×7fff6229c6e0: 0×0 0×0
[…]
0×7fff6229c8d0: 0×7fff6229c960 0×7fff622b49cd
0×7fff6229c8e0: 0×10269f05c 0×0
0×7fff6229c8f0: 0×7fff622c465c 0×7fff8a31e5c0 <_Z21dyldGlobalLockReleasev>
0×7fff6229c900: 0×7fff8fab99eb <abort> 0×10269f05c
0×7fff6229c910: 0×101000000000000 0×7fff622d2110
0×7fff6229c920: 0×7fff622d8d80 0×10269f078
0×7fff6229c930: 0×7fff622daac8 0×18
0×7fff6229c940: 0×0 0×0
0×7fff6229c950: 0×10269e030 0×0
0×7fff6229c960: 0×7fff6229c980 0×7fff622a1922
0×7fff6229c970: 0×0 0×0
0×7fff6229c980: 0×7fff6229ca50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6229c990: 0×1 0×7fff6229cb00
0×7fff6229c9a0: 0×7fff6229cb10 0xe223ea612ddc10b7
0×7fff6229c9b0: 0×8 0×0
0×7fff6229c9c0: 0xe223ea612ddc10b7 0×0
0×7fff6229c9d0: 0×0 0×0
0×7fff6229c9e0: 0×585f5f00474e414c 0×20435058005f4350
0×7fff6229c9f0: 0×0 0×0
0×7fff6229ca00: 0×0 0×0
0×7fff6229ca10: 0×0 0×0
0×7fff6229ca20: 0×0 0×0
0×7fff6229ca30: 0×7fff6229ca60 0×7fff8fab9a9c <abort+177>
0×7fff6229ca40: 0×0 0×0
0×7fff6229ca50: 0×7fffffffffdf 0×0
0×7fff6229ca60: 0×7fff6229ca70 0×10269dc29 <bar_5+9>
0×7fff6229ca70: 0×7fff6229ca80 0×10269dc39 <bar_4+9>
0×7fff6229ca80: 0×7fff6229ca90 0×10269dc49 <bar_3+9>
0×7fff6229ca90: 0×7fff6229caa0 0×10269dc59 <bar_2+9>
0×7fff6229caa0: 0×7fff6229cab0 0×10269dc69 <bar_1+9>
0×7fff6229cab0: 0×7fff6229cac0 0×10269dc79 <bar+9>
0×7fff6229cac0: 0×7fff6229cae0 0×10269dca0 <main+32>
0×7fff6229cad0: 0×7fff6229cb00 0×1
0×7fff6229cae0: 0×7fff6229caf0 0×10269db34 <start+52>

0×7fff6229caf0: 0×0 0×1
0×7fff6229cb00: 0×7fff6229cc48 0×0
0×7fff6229cb10: 0×7fff6229ccae 0×7fff6229ccca
[…]

Here’s the source code of the modeling application:

#define def_call(name,x,y) void name##_##x() { name##_##y(); }

#define def_final(name,x) void name##_##x() { }

#define def_final_abort(name,x) void name##_##x() { abort(); }

#define def_init(name,y) void name() { name##_##y(); }

#define def_init_alloc(name,y,size) void name() { int arr[size]; name##_##y(); *arr=0; }

def_final(foo,9)

def_call(foo,8,9)

def_call(foo,7,8)

def_call(foo,6,7)

def_call(foo,5,6)

def_call(foo,4,5)

def_call(foo,3,4)

def_call(foo,2,3)

def_call(foo,1,2)

def_init_alloc(foo,1,256)

def_final_abort(bar,5)

def_call(bar,4,5)

def_call(bar,3,4)

def_call(bar,2,3)

def_call(bar,1,2)

def_init(bar,1)

int main(int argc, const char * argv[])

{

    foo();

    bar();

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 23a, Mac OS X)

Tuesday, May 29th, 2012

This is a Mac OS X / GDB counterpart to Double Free (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e13f84c in free ()
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)

(gdb) x/2i 0x00000001035a8ef4-8
0x1035a8eec : mov -0×20(%rbp),%edi
0×1035a8eef : callq 0×1035a8f06

(gdb) frame 3
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)
at .../DoubleFree/main.c:23
23 free(p2);
Current language: auto; currently minimal

(gdb) x/g $rbp-0x20
0x7fff631a7ae0: 0x00007fe6a8801400

(gdb) x/2w 0x00007fe6a8801400
0x7fe6a8801400: 0x00000000 0xb0000000

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    free(p2);

    free(p1);

    free(p2);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 2, Mac OS X)

Sunday, May 27th, 2012

This is a Mac OS X / GDB counterpart to Dynamic Memory Corruption (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e1024ac in szone_error ()
#3 0x00007fff8e1024e8 in free_list_checksum_botch ()
#4 0x00007fff8e102a7b in small_free_list_remove_ptr ()
#5 0x00007fff8e106bf7 in szone_free_definite_size ()
#6 0x00007fff8e13f789 in free ()
#7 0x000000010afafe23 in main (argc=1, argv=0x7fff6abaeb08)

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    char *p3 = (char *) malloc (1024);

    printf(“p3 = %p\n”, p3);

 

    char *p4 = (char *) malloc (1024);

    printf(“p4 = %p\n”, p4);

 

    char *p5 = (char *) malloc (1024);

    printf(“p5 = %p\n”, p5);

 

    char *p6 = (char *) malloc (1024);

    printf(“p6 = %p\n”, p6);

 

    char *p7 = (char *) malloc (1024);

    printf(“p7 = %p\n”, p7);

 

    free(p6);

    free(p4);

    free(p2);

 

    printf(“Hello Crash!\n”);        

    strcpy(p2, “Hello Crash!”);

    strcpy(p4, “Hello Crash!”);

    strcpy(p6, “Hello Crash!”);

 

    p2 = (char *) malloc (512);

    printf(“p2 = %p\n”, p2);

 

    p4 = (char *) malloc (1024);

    printf(“p4 = %p\n”, p4);

 

    p6 = (char *) malloc (512);

    printf(“p6 = %p\n”, p6);

 

    free (p7);

    free (p6);

    free (p5);

    free (p4);

    free (p3);

    free (p2);

    free (p1);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 14, Mac OS X)

Wednesday, May 9th, 2012

This is a Mac OS X / GDB counterpart to Spiking Thread pattern previously described for Windows platforms:

(gdb) info threads
4 0×00007fff85b542df in sqrt$fenv_access_off ()
3 0×00007fff8616ee42 in __semwait_signal ()
2 0×00007fff8616ee42 in __semwait_signal ()
* 1 0×00007fff8616ee42 in __semwait_signal ()

We notice a non-waiting thread and switch to it:

(gdb) thread 4
[Switching to thread 4 (core thread 3)]
0x00007fff85b542df in sqrt$fenv_access_off ()

(gdb) bt
#0  0x00007fff85b542df in sqrt$fenv_access_off ()
#1  0×000000010cc85dc9 in thread_three (arg=0×7fff6c884ac0)
#2  0×00007fff8fac68bf in _pthread_start ()
#3  0×00007fff8fac9b75 in thread_start ()

If we disassemble the return address for thread_three function to come back from sqrt call we see an infinite loop:

(gdb) disass 0x000000010cc85dc9
Dump of assembler code for function thread_three:
0x000000010cc85db0 <thread_three+0>: push   %rbp
0×000000010cc85db1 <thread_three+1>: mov    %rsp,%rbp
0×000000010cc85db4 <thread_three+4>: sub    $0×10,%rsp
0×000000010cc85db8 <thread_three+8>: mov    %rdi,-0×10(%rbp)
0×000000010cc85dbc <thread_three+12>: mov    -0×10(%rbp),%ax
0×000000010cc85dc0 <thread_three+16>: movsd  (%rax),%xmm0
0×000000010cc85dc4 <thread_three+20>: callq  0×10cc85eac <dyld_stub_sqrt>
0×000000010cc85dc9 <thread_three+25>: mov    -0×10(%rbp),%rax
0×000000010cc85dcd <thread_three+29>: movsd  %xmm0,(%rax)
0×000000010cc85dd1 <thread_three+33>: jmpq   0×10cc85dbc <thread_three+12>
End of assembler dump.

Here’s the source code of the modeling application:

void * thread_one (void *arg)

{

    while (1)

    {

       sleep (1);

    }

 

    return 0;

}

 

void * thread_two (void *arg)

{

    while (1)

    {

        sleep (2);

    }

 

    return 0;

}

 

void * thread_three (void *arg)

{

    while (1)

    {

        *(double*)arg=sqrt(*(double *)arg);

    }

 

    return 0;

}

 

int main(int argc, const char * argv[])

{

    pthread_t threadID_one, threadID_two, threadID_three;

 

    double result = 0xffffffff;

 

    pthread_create (&threadID_one, NULL, thread_one, NULL);

    pthread_create (&threadID_two, NULL, thread_two, NULL);

    pthread_create (&threadID_three, NULL, thread_three,

       &result);

 

    pthread_join(threadID_three, NULL);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 6a, Mac OS X)

Thursday, May 3rd, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (code) pattern previously described for Windows platforms:

(gdb) bt
#0 0×0000000000000000 in ?? ()
#1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) disass 0×000000010e8cce73-3 0×000000010e8cce73
Dump of assembler code from 0×10e8cce70 to 0×10e8cce73:
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

(gdb) info r rdi
rdi 0x7fff6e4cbac0 140735043910336

(gdb) x/2 0x7fff6e4cbac0
0x7fff6e4cbac0: 0x0000000a 0×00000000

(gdb) p/x *($rdi+8)
$7 = 0×0

(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) ptype MYSTRUCT
type = struct _MyStruct_tag {
int data;
PFUNC pfunc;
}

(gdb) print {MYSTRUCT}0×7fff6e4cbac0
$2 = {data = 10, pfunc = 0}

Here’s the source code of the modeling application:

typedef void (*PFUNC)(void);

 

typedef struct _MyStruct_tag

{

    int   data;

    PFUNC pfunc;

} MYSTRUCT;

 

void bar(MYSTRUCT *ps)

{

    ps->pfunc();

}

 

void foo(MYSTRUCT *ps)

{

    bar(ps);

}

 

int main(int argc, const char * argv[])

{

    MYSTRUCT pstruct = {10, NULL};

 

    foo(&pstruct);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

GDB for WinDbg Users (Part 8)

Monday, April 30th, 2012

As we started providing memory dump analysis pattern examples for Mac OS X we resume our table of command correspondence between WinDbg and GDB providing some corrections on the way. For example, in the previous version of table we omitted a correspondence to ub WinDbg command. Now we provide such an equivalent:

(gdb) bt
[...]
#1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
[…]

(gdb) disas 0×000000010e8cce73-10 0×000000010e8cce73
Dump of assembler code from 0×10e8cce69 to 0×10e8cce73:
0×000000010e8cce69 : mov %edi,-0×8(%rbp)
0×000000010e8cce6c : mov -0×8(%rbp),%rdi
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

Please note that the beginning of assembly will be dependent on how good we guessed the offset:

(gdb) disas 0x000000010e8cce73-0×10 0×000000010e8cce73
Dump of assembler code from 0×10e8cce63 to 0×10e8cce73:
0×000000010e8cce63 : in $0×48,%eax
0×000000010e8cce65 : sub $0×10,%esp
0×000000010e8cce68 : mov %rdi,-0×8(%rbp)
0×000000010e8cce6c : mov -0×8(%rbp),%rdi
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

(gdb) disas 0x000000010e8cce73-0×13 0×000000010e8cce73
Dump of assembler code from 0×10e8cce60 to 0×10e8cce73:
0×000000010e8cce60 : push %rbp
0×000000010e8cce61 : mov %rsp,%rbp
0×000000010e8cce64 : sub $0×10,%rsp

0×000000010e8cce68 : mov %rdi,-0×8(%rbp)
0×000000010e8cce6c : mov -0×8(%rbp),%rdi
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

However, we can ignore that because our goal is to check whether a CPU instruction before a return address is a call.

Additional commands we add are x/<N>bc for db (WinDbg), thread <N> for ~<N>s (WinDbg, process dumps), maintenance info sections for for !address (WinDbg), add-symbol-file for .reload (WinDbg), info r for r (WinDbg).

Action                      | GDB                 | WinDbg
----------------------------------------------------------------
Start the process           | run                 | g
Exit                        | (q)uit              | q
Disassemble (forward)       | (disas)semble       | uf, u
Disassemble N instructions  | x/<N>i              | -
Disassemble (backward)      | disas <a-o> <a>     | ub
Stack trace                 | backtrace (bt)      | k
Full stack trace            | bt full             | kv
Stack trace with parameters | bt full             | kP
Partial trace (innermost)   | bt <N>              | k <N>
Partial trace (outermost)   | bt -<N>             | -
Stack trace for all threads | thread apply all bt | ~*k
Breakpoint                  | break               | bp
Frame numbers               | any bt command      | kn
Select frame                | frame               | .frame
Display parameters          | info args           | dv /t /i /V
Display locals              | info locals         | dv /t /i /V
Dump byte char array        | x/<N>bc             | db
Switch to thread            | thread <N>          | ~<N>s
Sections/regions            | maint info sections | !address
Load symbol file            | add-symbol-file     | .reload
CPU registers               | i(nfo) r            | r

Now an advertisement command:

(gdb) info training
(gdb) Accelerated Mac OS X Core Dump Analysis training

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 110, Mac OS X)

Sunday, April 22nd, 2012

This is a provisional Mac OS X example of Shared Buffer Overwrite pattern. Originally I wanted to construct a default C runtime heap corruption example using malloc / free functions. Unfortunately I couldn’t get heap corrupted easily as was possible in Windows Visual C++ environment by writing before or after allocated block. Desperately I printed allocated pointers and they all pointed to memory blocks laid out one after another without any headers in between (could be just a default Apple LLVM C runtime implementation and I have to check that with GCC). Therefore, any subsequent reallocation didn’t cause corruption either. So all this naturally fits into shared buffer overwrites or underwrites where corruption is only detectable when the overwritten data is used such as a pointer dereference.

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    strcpy(p1, “Hello World!”);

 

    printf(“p1 = %p\n”, p1);

    printf(“*p1 = %s\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    strcpy(p2, “Hello World!”);

 

    printf(“p2 = %p\n”, p2);

    printf(“*p2 = %s\n”, p2);

 

    char *p3 = (char *) malloc (1024);

    strcpy(p3, “Hello World!”);

 

    printf(“p3 = %p\n”, p3);

    printf(“*p3 = %s\n”, p3);

 

    strcpy(p2-sizeof(p2), “Hello Crash!”);

    strcpy(p3-sizeof(p3), “Hello Crash!”);

 

    p2 = (char *)realloc(p2, 2048);

 

    printf(“p2 = %p\n”, p2);

    printf(“*p2 = %s\n”, p2);

 

    char *p4 = (char *) malloc (1024);

    strcpy(p4-sizeof(p4), “Hello Crash!”);

 

    printf(“p4 = %p\n”, p4);

    printf(“*p4 = %s\n”, p4);

 

    p3 = (char *)realloc(p3, 2048);

 

    printf(“p3 = %p\n”, p3);

    printf(“*p3 = %s\n”, p3);

 

    char *p5 = NULL; // to force a core dump

    *p5 = 0;

 

    free (p4);

    free (p3);

    free (p2);

    free (p1);

 

    return 0;

}

When we run the program above we get this output:

p1 = 0x7fc6d9000000
*p1 = Hello World!
p2 = 0×7fc6d9001400
*p2 = Hello World!
p3 = 0×7fc6d9001800
*p3 = Hello World!
p2 = 0×7fc6d9001c00
*p2 = ash!
p4 = 0×7fc6d9001400
*p4 = ash!
p3 = 0×7fc6d9002400
*p3 = ash!
Segmentation fault: 11 (core dumped)

Now is GDB output:

(gdb) x/1024bc p1
0x7fc6d9000000: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9000008: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9000010: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90003e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90003f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/32bc p1+1024-sizeof(p1)
0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9000400: 42 ‘*’ 112 ‘p’ 51 ‘3′ 32 ‘ ‘ 61 ‘=’ 32 ‘ ‘ 97 ‘a’ 115 ’s’
0×7fc6d9000408: 104 ‘h’ 33 ‘!’
10 ‘\n’ 100 ‘d’ 57 ‘9′ 48 ‘0′ 48 ‘0′ 50 ‘2′
0×7fc6d9000410: 52 ‘4′ 48 ‘0′ 48 ‘0′ 10 ‘\n’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/2048bc p2
0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d9001fe8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001ff0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001ff8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9002000: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002008: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90023e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90023f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p2-sizeof(p2)
0×7fc6d9001bf8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c18: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c20: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c28: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c30: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p2+2048-sizeof(p2)
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/1024bc p3
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90027e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90027f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p3-sizeof(p3)
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p3+1024-sizeof(p3)
0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002800: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002808: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/1024bc p4
0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90017e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90017f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
(gdb) x/64bc p4-sizeof(p4)
0×7fc6d90013f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′
32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p4+1024-sizeof(p4)
0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9001800: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′
32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001808: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′
0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming 2nd edition of Memory Dump Analysis Anthology, Volume 1

Sunday, April 15th, 2012

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

  • Title: Memory Dump Analysis Anthology, Volume 1
  • Author: Dmitry Vostokov
  • Publisher: OpenTask (Summer 2012)
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 800 pages
  • ISBN-13: 978-1-908043-35-1
  • Hardcover: 800 pages
  • ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 1, Mac OS X)

Wednesday, April 4th, 2012

The first Windows pattern called Multiple Exceptions in user mode now has Mac OS X equivalent. In the example below there are 3 threads and two of them experienced NULL Pointer (data) access violation exception:

(gdb) thread apply all bt full

Thread 3 (core thread 2):
#0  0x00000001062ffe4e in thread_two (arg=0x0)
at main.c:24
p = (int *) 0×0
#1  0×00007fff8abf58bf in _pthread_start ()
No symbol table info available.
#2  0×00007fff8abf8b75 in thread_start ()
No symbol table info available.

Thread 2 (core thread 1):
#0  0x00000001062ffe1e in thread_one (arg=0x0)
at main.c:16
p = (int *) 0×0
#1  0×00007fff8abf58bf in _pthread_start ()
No symbol table info available.
#2  0×00007fff8abf8b75 in thread_start ()
No symbol table info available.

Thread 1 (core thread 0):
#0  0x00007fff854e0e42 in __semwait_signal ()
No symbol table info available.
#1  0x00007fff8ababdea in nanosleep ()
No symbol table info available.
#2  0x00007fff8ababc2c in sleep ()
No symbol table info available.
#3  0x00000001062ffec3 in main (argc=1, argv=0x7fff65efeab8)
at main.c:36
threadID_one = (pthread_t) 0×1063b4000
threadID_two = (pthread_t) 0×106581000

(gdb) thread 2
[Switching to thread 2 (core thread 1)]
0x00000001062ffe1e in thread_one (arg=0x0)
at main.c:16
16    *p = 1;

(gdb) p/x p
$1 = 0×0

(gdb) thread 3
[Switching to thread 3 (core thread 2)]
0x00000001062ffe4e in thread_two (arg=0x0)
at main.c:24
24    *p = 2;

(gdb) p/x p
$2 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 25, Mac OS X)

Sunday, March 25th, 2012

This is a Mac OS X / GDB counterpart to Stack Trace pattern previously described for Windows platforms. Here we show a stack trace when symbols are not available and also how to apply symbols:

(gdb) bt
#0  0×000000010d3b0e90 in ?? ()
#1  0×000000010d3b0ea9 in ?? ()
#2  0×000000010d3b0ec4 in ?? ()
#3  0×000000010d3b0e74 in ?? ()

(gdb) maintenance info sections
Exec file:
[...]
Core file:
`/cores/core.262', file type mach-o-le.
0×000000010d3b0000->0×000000010d3b1000 at 0×00001000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b1000->0×000000010d3b2000 at 0×00002000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b2000->0×000000010d3b3000 at 0×00003000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b3000->0×000000010d3b4000 at 0×00004000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b4000->0×000000010d3b5000 at 0×00005000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b5000->0×000000010d3b6000 at 0×00006000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b6000->0×000000010d3cb000 at 0×00007000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3cb000->0×000000010d3cc000 at 0×0001c000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3cc000->0×000000010d3cd000 at 0×0001d000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3cd000->0×000000010d3e2000 at 0×0001e000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3e2000->0×000000010d3e3000 at 0×00033000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3e3000->0×000000010d3e4000 at 0×00034000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d400000->0×000000010d500000 at 0×00035000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
[…]

(gdb) add-symbol-file ~/Documents/Work/Test.sym 0×000000010d3b0000
add symbol table from file “/Users/DumpAnalysis/Documents/Work/Test.sym” at
LC_SEGMENT.__TEXT = 0×10d3b0000
(y or n) y
Reading symbols from /Users/DumpAnalysis/Documents/Work/Test.sym…done.

(gdb) bt
#0  0x000000010d3b0e90 in bar () at main.c:15
#1  0x000000010d3b0ea9 in foo () at main.c:20
#2  0x000000010d3b0ec4 in main (argc=1,
argv=0x7fff6cfafbf8) at main.c:25

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

GDB Annoyances: Incomplete Stack Trace

Sunday, March 25th, 2012

Users of WinDbg debugger accustomed to full thread stack traces will wonder whether a thread starts from main:

(gdb) where
#0  0x000000010d3b0e90 in bar () at main.c:15
#1  0x000000010d3b0ea9 in foo () at main.c:20
#2  0x000000010d3b0ec4 in main (argc=1,
argv=0x7fff6cfafbf8) at main.c:25

Of course, not and by default a stack trace is shown starting from main function. You can change this behavior by using the following command:

(gdb) set backtrace past-main

Now we see an additional frame:

(gdb) where
#0  0x000000010d3b0e90 in bar () at main.c:15
#1  0x000000010d3b0ea9 in foo () at main.c:20
#2  0x000000010d3b0ec4 in main (argc=1,
argv=0x7fff6cfafbf8) at main.c:25
#3  0×000000010d3b0e74 in start ()

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 6b, Mac OS X)

Sunday, March 25th, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (data) pattern previously described for Windows platforms:

(gdb) bt
#0  0×000000010d3b0e90 in bar () at main.c:15
#1  0×000000010d3b0ea9 in foo () at main.c:20
#2  0×000000010d3b0ec4 in main (argc=1,
argv=0×7fff6cfafbf8) at main.c:25

(gdb) disassemble
Dump of assembler code for function bar:
0x000000010d3b0e80 <bar+0>: push   %rbp
0×000000010d3b0e81 <bar+1>: mov    %rsp,%rbp
0×000000010d3b0e84 <bar+4>: movq   $0×0,-0×8(%rbp)
0×000000010d3b0e8c <bar+12>: mov    -0×8(%rbp),%rax
0×000000010d3b0e90 <bar+16>: movl   $0×1,(%rax)
0×000000010d3b0e96 <bar+22>: pop    %bp
0×000000010d3b0e97 <bar+23>: retq
End of assembler dump.

(gdb) p/x $rax
$1 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis