Software Diagnostics Library

While preparing a seminar on Software Diagnostics I made a lot of notes and realized that a system of patterns, corresponding vocabulary and pattern language are needed for this discipline. Here patterns are supposed to be broad in nature and be different from patterns for specific artifacts such as memory dumps and software traces. So the first pattern addresses a diagnostic encounter with a First Fault in comparison to subsequent faults where the problem becomes noticeable and diagnostic resources are allocated. Such faults should not be dismissed. Dan Skwire is a passionate advocate of first fault software problem solving and wrote a book:

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

The following paper proposes distributed control flow reconstruction for first fault diagnosis:

TraceBack: First Fault Diagnosis by Reconstruction of Distributed Control Flow

Memory Dump Analysis Services uses patterns of abnormal software behavior for its first fault diagnostics that doesn’t require any special instrumentation:

Join Debugging Diagnostics Revolution!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Stack Trace Change is an important pattern for differential memory dump analysis, for example, when memory dumps were generated before and after a problem such as a CPU spike or hang. In the example below we have a normal expected thread stack trace from a memory dump saved before an application was reported unresponsive and another different thread stack trace after:

3  Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen
ChildEBP RetAddr
037dfadc 75210bdd ntdll!ZwWaitForMultipleObjects+0x15
037dfb78 75791a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
037dfbc0 7511086a kernel32!WaitForMultipleObjectsExImplementation+0xe0
037dfc14 00d17c1d user32!RealMsgWaitForMultipleObjectsEx+0x14d
037dfc3c 00ce161d ApplicationA!MsgWaitForMultipleObjects+0x2d
037dfc60 00cdc757 ApplicationA!WaitForSignal+0x1d
037dfc80 00cdaaf6 ApplicationA!WorkLoop+0x57
037dfca4 7579339a ApplicationA!ThreadStart+0x26
037dfcb0 77699ef2 kernel32!BaseThreadInitThunk+0xe
037dfcf0 77699ec5 ntdll!__RtlUserThreadStart+0x70
037dfd08 00000000 ntdll!_RtlUserThreadStart+0x1b

3  Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen
ChildEBP RetAddr
037df38c 752131bb ntdll!ZwDelayExecution+0x15
037df3f4 75213a8b KERNELBASE!SleepEx+0x65
037df404 00d1670b KERNELBASE!Sleep+0xf
037df40c 00d350ef ApplicationA!Sleep+0xb
037df430 6a868aab ApplicationA!PutData+0xbf
037df444 6a8662ec ModuleA!OutputData+0x1b
037df464 00d351de ModuleA!ProcessData+0x16c
037df4a4 00ca8cb4 ApplicationA!SendData+0xbe
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a Mac OS X / GDB counterpart to Coincidental Symbolic Information pattern previously described for Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:

(gdb) x $rsp
0x7fff6a162a38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6a162000
[...]
0x7fff6a162960: 0x7fff6a162980 0x7fff6a167922
0x7fff6a162970: 0x0 0x0
0x7fff6a162980: 0x7fff6a162a50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6a162990: 0×1 0×7fff6a162b00
0×7fff6a1629a0: 0×7fff6a162b10 0×7fff6a162bc0
0×7fff6a1629b0: 0×8 0×0
[…]
0×7fff6a162a00: 0×0 0×0
0×7fff6a162a10: 0×0 0×0
0×7fff6a162a20: 0×0 0×0
0×7fff6a162a30: 0×7fff6a162a60 0×7fff8fab9a9c <abort+177>
0×7fff6a162a40: 0×0 0×0
0×7fff6a162a50: 0×7fffffffffdf 0×0
[…]
0×7fff6a163040: 0×35000 0×0
0×7fff6a163050: 0×35000 0×500000007
0×7fff6a163060: 0×7 0×747865745f5f
0×7fff6a163070: 0×0 0×545845545f5f
0×7fff6a163080: 0×0 0×7fff5fc01000 <__dyld_stub_binding_helper>
0×7fff6a163090: 0×22c9d 0xc00001000
0×7fff6a1630a0: 0×0 0×80000400
[…]

(gdb) disass 0×7fff8a31e716
Dump of assembler code for function dyld_stub_binder_:
0×00007fff8a31e709 <dyld_stub_binder_+0>: mov 0×8(%rbp),%rdi
0×00007fff8a31e70d <dyld_stub_binder_+4>: mov 0×10(%rbp),%rsi
0×00007fff8a31e711 <dyld_stub_binder_+8>: callq 0×7fff8a31e86d <_Z21_dyld_fast_stub_entryPvl>
0×00007fff8a31e716 <dyld_stub_binder_+13>: mov %rax,%r11
0×00007fff8a31e719 <dyld_stub_binder_+16>: movdqa 0×40(%rsp),%xmm0
0×00007fff8a31e71f <dyld_stub_binder_+22>: movdqa 0×50(%rsp),%xmm1
0×00007fff8a31e725 <dyld_stub_binder_+28>: movdqa 0×60(%rsp),%xmm2
0×00007fff8a31e72b <dyld_stub_binder_+34>: movdqa 0×70(%rsp),%xmm3
0×00007fff8a31e731 <dyld_stub_binder_+40>: movdqa 0×80(%rsp),%xmm4
0×00007fff8a31e73a <dyld_stub_binder_+49>: movdqa 0×90(%rsp),%xmm5
0×00007fff8a31e743 <dyld_stub_binder_+58>: movdqa 0xa0(%rsp),%xmm6
0×00007fff8a31e74c <dyld_stub_binder_+67>: movdqa 0xb0(%rsp),%xmm7
0×00007fff8a31e755 <dyld_stub_binder_+76>: mov (%rsp),%rdi
0×00007fff8a31e759 <dyld_stub_binder_+80>: mov 0×8(%rsp),%rsi
0×00007fff8a31e75e <dyld_stub_binder_+85>: mov 0×10(%rsp),%rdx
0×00007fff8a31e763 <dyld_stub_binder_+90>: mov 0×18(%rsp),%rcx
0×00007fff8a31e768 <dyld_stub_binder_+95>: mov 0×20(%rsp),%r8
0×00007fff8a31e76d <dyld_stub_binder_+100>: mov 0×28(%rsp),%r9
0×00007fff8a31e772 <dyld_stub_binder_+105>: mov 0×30(%rsp),%rax
0×00007fff8a31e777 <dyld_stub_binder_+110>: add $0xc0,%rsp
0×00007fff8a31e77e <dyld_stub_binder_+117>: pop %rbp
0×00007fff8a31e77f <dyld_stub_binder_+118>: add $0×10,%rsp
0×00007fff8a31e783 <dyld_stub_binder_+122>: jmpq *%r11

(gdb) x/2i 0×7fff8fab9a9c
0×7fff8fab9a9c <abort+177>: mov $0×2710,%edi
0×7fff8fab9aa1 <abort+182>: callq 0×7fff8fab9c43 <usleep$nocancel>

(gdb) disass 0×7fff8fab9a9c-5 0×7fff8fab9a9c
Dump of assembler code from 0×7fff8fab9a97 to 0×7fff8fab9a9c:
0×00007fff8fab9a97 <abort+172>: callq 0×7fff8fb1f54a <dyld_stub_kill>
End of assembler dump.

(gdb) disass 0×7fff5fc01000
Dump of assembler code for function __dyld_stub_binding_helper:
0×00007fff5fc01000 <__dyld_stub_binding_helper+0>: add %al,(%rax)
0×00007fff5fc01002 <__dyld_stub_binding_helper+2>: add %al,(%rax)
0×00007fff5fc01004 <__dyld_stub_binding_helper+4>: add %al,(%rax)
0×00007fff5fc01006 <__dyld_stub_binding_helper+6>: add %al,(%rax)
End of assembler dump.

(gdb) x/10 0×7fff5fc01000-0×10
0×7fff5fc00ff0: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01000 <__dyld_stub_binding_helper>: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01010 <__dyld_offset_to_dyld_all_image_infos>: 0×00000000 0×00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis