Archive for June, 2012

Crash Dump Analysis Patterns (Part 178)

Wednesday, June 27th, 2012

One of the frequent problems is an access violation at an address that belongs to Unloaded Module. Here’s an example that recently happened on our machine during an auto-update of the popular software package so we immediately attached a debugger after seeing WER dialog box:

0:000> ~*k

.  0  Id: bc8.bcc Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0035f1c4 771a0bdd ntdll!ZwWaitForMultipleObjects+0x15
0035f260 75771a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
0035f2a8 75774208 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0035f2c4 757980a4 kernel32!WaitForMultipleObjects+0x18
0035f330 75797f63 kernel32!WerpReportFaultInternal+0x186
0035f344 75797858 kernel32!WerpReportFault+0x70
0035f354 757977d7 kernel32!BasepReportFault+0x20
0035f3e0 77ec74df kernel32!UnhandledExceptionFilter+0x1af
0035f3e8 77ec73bc ntdll!__RtlUserThreadStart+0x62
0035f3fc 77ec7261 ntdll!_EH4_CallFilterFunc+0x12
0035f424 77eab459 ntdll!_except_handler4+0x8e
0035f448 77eab42b ntdll!ExecuteHandler2+0x26
0035f46c 77eab3ce ntdll!ExecuteHandler+0x24
0035f4f8 77e60133 ntdll!RtlDispatchException+0x127
0035f4f8 73eb2200 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0035f844 76e462fa <Unloaded_fpb.tmp>+0×12200
0035f870 76e46d3a USER32!InternalCallWinProc+0×23
0035f8e8 76e4965e USER32!UserCallWinProcCheckWow+0×109
0035f92c 76e496c5 USER32!SendMessageWorker+0×581
0035f950 7269c05c USER32!SendMessageW+0×7f
0035f9ec 7270be62 comctl32!CCSendNotify+0xc19
0035fa28 75f6f52a comctl32!SendNotify+0×36
0035fa4c 75f61d66 SHELL32!SetAppStartingCursor+0×6d
0035fa64 75f61ee2 SHELL32!CShellExecute::ExecuteNormal+0×16
0035fa78 75f61e70 SHELL32!ShellExecuteNormal+0×33
0035fa90 75f53cd0 SHELL32!ShellExecuteExW+0×62
0035fae4 003e2211 SHELL32!ShellExecuteW+0×77
0035fbc4 77e838be InstallFlashPlayer+0×2211
0035fcb4 77e83492 ntdll!RtlpFreeHeap+0xbb1
0035fcd4 757714dd ntdll!RtlFreeHeap+0×142
0035fce8 003f0324 kernel32!HeapFree+0×14
0035fd80 003f0241 InstallFlashPlayer+0×10324
0035fe10 7577339a InstallFlashPlayer+0×10241
0035fe1c 77e89ef2 kernel32!BaseThreadInitThunk+0xe
0035fe5c 77e89ec5 ntdll!__RtlUserThreadStart+0×70
0035fe74 00000000 ntdll!_RtlUserThreadStart+0×1b

1  Id: bc8.6b0 Suspend: 2 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
03e1f9e0 77ea2f51 ntdll!ZwWaitForMultipleObjects+0x15
03e1fb74 7577339a ntdll!TppWaiterpThread+0x33d
03e1fb80 77e89ef2 kernel32!BaseThreadInitThunk+0xe
03e1fbc0 77e89ec5 ntdll!__RtlUserThreadStart+0x70
03e1fbd8 00000000 ntdll!_RtlUserThreadStart+0x1b

2  Id: bc8.8dc Suspend: 2 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
03f5fd50 77ea3352 ntdll!NtWaitForWorkViaWorkerFactory+0x12
03f5feb0 7577339a ntdll!TppWorkerThread+0x216
03f5febc 77e89ef2 kernel32!BaseThreadInitThunk+0xe
03f5fefc 77e89ec5 ntdll!__RtlUserThreadStart+0x70
03f5ff14 00000000 ntdll!_RtlUserThreadStart+0x1b

3  Id: bc8.944 Suspend: 2 Teb: 7efaf000 Unfrozen
ChildEBP RetAddr
0416f8b4 77ea3352 ntdll!NtWaitForWorkViaWorkerFactory+0x12
0416fa14 7577339a ntdll!TppWorkerThread+0x216
0416fa20 77e89ef2 kernel32!BaseThreadInitThunk+0xe
0416fa60 77e89ec5 ntdll!__RtlUserThreadStart+0x70
0416fa78 00000000 ntdll!_RtlUserThreadStart+0x1b

Exception thread shows fpb.tmp module as unloaded:

0:000> lmv m fpb.tmp
start    end        module name

Unloaded modules:
00cb0000 00d5a000   fpb.tmp

Timestamp: Fri Jun 01 02:56:00 2012 (4FC82130)
Checksum:  000B0CD5
ImageSize:  000AA000
73ea0000 73f15000   fpb.tmp
Timestamp: Fri Jun 01 02:49:25 2012 (4FC81FA5)
Checksum:  0007A7CE
ImageSize:  00075000

We change the exception thread context to get registers at the time of the exception:

0:000> kv
ChildEBP RetAddr  Args to Child
0035f1c4 771a0bdd 00000002 0035f214 00000001 ntdll!ZwWaitForMultipleObjects+0x15
0035f260 75771a2c 0035f214 0035f288 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
0035f2a8 75774208 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0035f2c4 757980a4 00000002 0035f2f8 00000000 kernel32!WaitForMultipleObjects+0x18
0035f330 75797f63 0035f410 00000001 00000001 kernel32!WerpReportFaultInternal+0x186
0035f344 75797858 0035f410 00000001 0035f3e0 kernel32!WerpReportFault+0x70
0035f354 757977d7 0035f410 00000001 658587c7 kernel32!BasepReportFault+0x20
0035f3e0 77ec74df 00000000 77ec73bc 00000000 kernel32!UnhandledExceptionFilter+0x1af
0035f3e8 77ec73bc 00000000 0035fe5c 77e7c530 ntdll!__RtlUserThreadStart+0x62
0035f3fc 77ec7261 00000000 00000000 00000000 ntdll!_EH4_CallFilterFunc+0x12
0035f424 77eab459 fffffffe 0035fe4c 0035f560 ntdll!_except_handler4+0x8e
0035f448 77eab42b 0035f510 0035fe4c 0035f560 ntdll!ExecuteHandler2+0x26
0035f46c 77eab3ce 0035f510 0035fe4c 0035f560 ntdll!ExecuteHandler+0x24
0035f4f8 77e60133 0135f510 0035f560 0035f510 ntdll!RtlDispatchException+0x127
0035f4f8 73eb2200 0135f510 0035f560 0035f510 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0035f560)
WARNING: Frame IP not in any known module. Following frames may be wrong.
0035f844 76e462fa 000201ce 0000004e 00000000 <Unloaded_fpb.tmp>+0×12200
0035f870 76e46d3a 73eb2200 000201ce 0000004e USER32!InternalCallWinProc+0×23
0035f8e8 76e4965e 00000000 73eb2200 000201ce USER32!UserCallWinProcCheckWow+0×109
0035f92c 76e496c5 013907f0 00000000 73eb2200 USER32!SendMessageWorker+0×581
0035f950 7269c05c 000201ce 0000004e 00000000 USER32!SendMessageW+0×7f
0035f9ec 7270be62 0035fa00 fffffff7 00000000 comctl32!CCSendNotify+0xc19
0035fa28 75f6f52a 000201ce 00000000 fffffff7 comctl32!SendNotify+0×36
0035fa4c 75f61d66 000201ce 00000001 00001500 SHELL32!SetAppStartingCursor+0×6d
0035fa64 75f61ee2 0035faa4 00001500 0035faa4 SHELL32!CShellExecute::ExecuteNormal+0×16
0035fa78 75f61e70 0035faa4 00001500 00000200 SHELL32!ShellExecuteNormal+0×33
0035fa90 75f53cd0 0035faa4 003fb654 003fa554 SHELL32!ShellExecuteExW+0×62
0035fae4 003e2211 000201ce 003fa554 0035fb14 SHELL32!ShellExecuteW+0×77
0035fbc4 77e838be 00da0138 77e8389a 77c467ad InstallFlashPlayer+0×2211
0035fcb4 77e83492 00000000 00da2320 00da2320 ntdll!RtlpFreeHeap+0xbb1
0035fcd4 757714dd 00da0000 00000000 00da2320 ntdll!RtlFreeHeap+0×142
0035fce8 003f0324 00da0000 00000000 003f0343 kernel32!HeapFree+0×14
0035fd80 003f0241 003e0000 00000000 010d3135 InstallFlashPlayer+0×10324
0035fe10 7577339a 7efde000 0035fe5c 77e89ef2 InstallFlashPlayer+0×10241
0035fe1c 77e89ef2 7efde000 77c46545 00000000 kernel32!BaseThreadInitThunk+0xe
0035fe5c 77e89ec5 003f02ac 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0035fe74 00000000 003f02ac 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> .cxr 0035f560
eax=73eb2200 ebx=00000000 ecx=01080d68 edx=00000000 esi=73eb2200 edi=00000000
eip=73eb2200 esp=0035f848 ebp=0035f870 iopl=0 nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b efl=00210206
<Unloaded_fpb.tmp>+0×12200:
73eb2200 ??              ???

Then we double check that a window procedure was indeed called from that module range:

0:000> kv
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0035f844 76e462fa 000201ce 0000004e 00000000 <Unloaded_fpb.tmp>+0×12200
0035f870 76e46d3a 73eb2200 000201ce 0000004e USER32!InternalCallWinProc+0×23
0035f8e8 76e4965e 00000000 73eb2200 000201ce USER32!UserCallWinProcCheckWow+0×109
0035f92c 76e496c5 013907f0 00000000 73eb2200 USER32!SendMessageWorker+0×581
0035f950 7269c05c 000201ce 0000004e 00000000 USER32!SendMessageW+0×7f
0035f9ec 7270be62 0035fa00 fffffff7 00000000 comctl32!CCSendNotify+0xc19
0035fa28 75f6f52a 000201ce 00000000 fffffff7 comctl32!SendNotify+0×36
0035fa4c 75f61d66 000201ce 00000001 00001500 SHELL32!SetAppStartingCursor+0×6d
0035fa64 75f61ee2 0035faa4 00001500 0035faa4 SHELL32!CShellExecute::ExecuteNormal+0×16
0035fa78 75f61e70 0035faa4 00001500 00000200 SHELL32!ShellExecuteNormal+0×33
0035fa90 75f53cd0 0035faa4 003fb654 003fa554 SHELL32!ShellExecuteExW+0×62
0035fae4 003e2211 000201ce 003fa554 0035fb14 SHELL32!ShellExecuteW+0×77
0035fbc4 77e838be 00da0138 77e8389a 77c467ad InstallFlashPlayer+0×2211
0035fcb4 77e83492 00000000 00da2320 00da2320 ntdll!RtlpFreeHeap+0xbb1
00da15a0 00000000 00da1780 02971450 003e1000 ntdll!RtlFreeHeap+0×142

0:000> ub 76e462fa
USER32!InternalCallWinProc+0×6:
76e462dd 68cdabbadc      push    0DCBAABCDh
76e462e2 56              push    esi
76e462e3 ff7518          push    dword ptr [ebp+18h]
76e462e6 ff7514          push    dword ptr [ebp+14h]
76e462e9 ff7510          push    dword ptr [ebp+10h]
76e462ec ff750c          push    dword ptr [ebp+0Ch]
76e462ef 64800dca0f000001 or      byte ptr fs:[0FCAh],1
76e462f7 ff5508          call    dword ptr [ebp+8]

We now get a memory value pointed to by EBP+8 address:

0:000> r
Last set context:
eax=73eb2200 ebx=00000000 ecx=01080d68 edx=00000000 esi=73eb2200 edi=00000000
eip=73eb2200 esp=0035f848 ebp=0035f870 iopl=0 nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00210206
<Unloaded_fpb.tmp>+0×12200:
73eb2200 ??              ???

0:000> dp 0035f870+8 l1
0035f878  73eb2200

0:000> dd 73eb2200
73eb2200  ???????? ???????? ???????? ????????
73eb2210  ???????? ???????? ???????? ????????
73eb2220  ???????? ???????? ???????? ????????
73eb2230  ???????? ???????? ???????? ????????
73eb2240  ???????? ???????? ???????? ????????
73eb2250  ???????? ???????? ???????? ????????
73eb2260  ???????? ???????? ???????? ????????
73eb2270  ???????? ???????? ???????? ????????

The value is indeed belongs to unloaded fpb.tmp module address range:

0:000> ln 73eb2200
(73eb2200)   <Unloaded_fpb.tmp>+0×12200

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

FBI (Debugging Slang, Part 35)

Wednesday, June 27th, 2012

FBI - Fighting Bugs Inside.

Examples: I’m doing an FBI work now!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

poo (Debugging Slang, Part 34)

Monday, June 25th, 2012

poo - a function that follows foo and bar with a purpose to trigger a crash event, a breakpoint or save memory state.

Examples: void main() { foo(); } void foo() { poo(); } void poo() { asm int 3; }

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

STaMPs (Debugging Slang, Part 33)

Monday, June 25th, 2012

STaMPs - Software Trace and Memory Patterns. Stack Trace and Memory Patterns.

Examples: Got a few visible stamps on this trace. And more stamps on that crash dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Webinar: Introduction to Systemic Software Diagnostics

Sunday, June 24th, 2012

This is a second Webinar from Memory Dump Analysis Services on software diagnostics. The first one is about pattern recognition. During this Webinar you will learn how to apply systems theory and systems thinking for effective and efficient abnormal software behavior diagnostics: the foundation of software troubleshooting and debugging. The seminar summarizes 6 years of research done by Software Diagnostics Institute started with a short blog post Dumps and Systems Theory.

 Introduction to Systemic Software Diagnostics Logo

Title: Introduction to Systemic Software Diagnostics: Systems Thinking in Memory Dump and Software Trace Analysis
Date: 3rd of September, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/377382766

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Webinar on Victimware: The Missing Part of the Equation

Sunday, June 24th, 2012

Memory Dump Analysis Services organizes a free webinar on a unified malware and victimware analysis by using behavioral and structural patterns including a live memory dump analysis example.

Victimware Analysis Webinar Logo

Date: 2nd of July, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/332458406

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 51)

Saturday, June 23rd, 2012

Counter Value pattern covers performance monitoring and its logs. A counter value is some variable in memory, for example, a module variable, that is updated periodically to reflect some aspect of state or it can be calculated from different such variables and presented in trace messages. Such messages can also be organized in a similar format as ETW based traces we usually consider as examples for our trace patterns:

Source  PID TID   Function         Value
=================================================
[…]
System    0   0   Committed Memory 12,002,234,654
Process 844   0   Private Bytes    345,206,456
System    0   0   Committed Memory 12,002,236,654
Process 844   0   Working Set      122,160,068
[…]

Therefore, all other trace patterns such as adjoint thread (can be visualized via different colors on a graph), focus of tracing, characteristic message block (for graphs), activity regionsignificant event, and others can be applicable here. There are also some specific patterns such as global monotonicity and constant value that we discuss with examples in subsequent parts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Individual and Enterprise Software Diagnostics Certifications

Monday, June 18th, 2012

Memory Dump Analysis Services will be administering certifications developed by Software Diagnostics Institute for memory dump and software trace analysis:

Software Diagnostics Maturity Enterprise Certification
Memory Dump Analysis Certification is available this September

Debugging TV Frames episode 0×10 contains some background information.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Software Diagnostics Institute

Tuesday, June 12th, 2012

DumpAnalysis.org portal has been reorganized to Software Diagnostics Institute to reflect the nature of its research activities. More updates later on.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A NoSQL Problem (Debugging Slang, Part 32)

Sunday, June 10th, 2012

A NoSQL Problem - when nothing appears on a refresh.

Examples: I got a NoSQL problem when I signed in to that social website.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Patterns of Software Diagnostics (Part 1)

Saturday, June 9th, 2012

While preparing a seminar on Software Diagnostics I made a lot of notes and realized that a system of patterns, corresponding vocabulary and pattern language are needed for this discipline. Here patterns are supposed to be broad in nature and be different from patterns for specific artifacts such as memory dumps and software traces. So the first pattern addresses a diagnostic encounter with a First Fault in comparison to subsequent faults where the problem becomes noticeable and diagnostic resources are allocated. Such faults should not be dismissed. Dan Skwire is a passionate advocate of first fault software problem solving and wrote a book:

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

The following paper proposes distributed control flow reconstruction for first fault diagnosis:

TraceBack: First Fault Diagnosis by Reconstruction of Distributed Control Flow

Memory Dump Analysis Services uses patterns of abnormal software behavior for its first fault diagnostics that doesn’t require any special instrumentation:

Join Debugging Diagnostics Revolution!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 177)

Saturday, June 9th, 2012

Stack Trace Change is an important pattern for differential memory dump analysis, for example, when memory dumps were generated before and after a problem such as a CPU spike or hang. In the example below we have a normal expected thread stack trace from a memory dump saved before an application was reported unresponsive and another different thread stack trace after:

3  Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen
ChildEBP RetAddr
037dfadc 75210bdd ntdll!ZwWaitForMultipleObjects+0x15
037dfb78 75791a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
037dfbc0 7511086a kernel32!WaitForMultipleObjectsExImplementation+0xe0
037dfc14 00d17c1d user32!RealMsgWaitForMultipleObjectsEx+0x14d
037dfc3c 00ce161d ApplicationA!MsgWaitForMultipleObjects+0x2d
037dfc60 00cdc757 ApplicationA!WaitForSignal+0x1d
037dfc80 00cdaaf6 ApplicationA!WorkLoop+0x57
037dfca4 7579339a ApplicationA!ThreadStart+0x26
037dfcb0 77699ef2 kernel32!BaseThreadInitThunk+0xe
037dfcf0 77699ec5 ntdll!__RtlUserThreadStart+0x70
037dfd08 00000000 ntdll!_RtlUserThreadStart+0x1b

3  Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen
ChildEBP RetAddr
037df38c 752131bb ntdll!ZwDelayExecution+0x15
037df3f4 75213a8b KERNELBASE!SleepEx+0x65
037df404 00d1670b KERNELBASE!Sleep+0xf
037df40c 00d350ef ApplicationA!Sleep+0xb
037df430 6a868aab ApplicationA!PutData+0xbf
037df444 6a8662ec ModuleA!OutputData+0x1b
037df464 00d351de ModuleA!ProcessData+0x16c
037df4a4 00ca8cb4 ApplicationA!SendData+0xbe
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 24, Mac OS X)

Saturday, June 9th, 2012

This is a Mac OS X / GDB counterpart to Coincidental Symbolic Information pattern previously described for Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:

(gdb) x $rsp
0x7fff6a162a38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6a162000
[...]
0x7fff6a162960: 0x7fff6a162980 0x7fff6a167922
0x7fff6a162970: 0x0 0x0
0x7fff6a162980: 0x7fff6a162a50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6a162990: 0×1 0×7fff6a162b00
0×7fff6a1629a0: 0×7fff6a162b10 0×7fff6a162bc0
0×7fff6a1629b0: 0×8 0×0
[…]
0×7fff6a162a00: 0×0 0×0
0×7fff6a162a10: 0×0 0×0
0×7fff6a162a20: 0×0 0×0
0×7fff6a162a30: 0×7fff6a162a60 0×7fff8fab9a9c <abort+177>
0×7fff6a162a40: 0×0 0×0
0×7fff6a162a50: 0×7fffffffffdf 0×0
[…]
0×7fff6a163040: 0×35000 0×0
0×7fff6a163050: 0×35000 0×500000007
0×7fff6a163060: 0×7 0×747865745f5f
0×7fff6a163070: 0×0 0×545845545f5f
0×7fff6a163080: 0×0 0×7fff5fc01000 <__dyld_stub_binding_helper>
0×7fff6a163090: 0×22c9d 0xc00001000
0×7fff6a1630a0: 0×0 0×80000400
[…]

(gdb) disass 0×7fff8a31e716
Dump of assembler code for function dyld_stub_binder_:
0×00007fff8a31e709 <dyld_stub_binder_+0>: mov 0×8(%rbp),%rdi
0×00007fff8a31e70d <dyld_stub_binder_+4>: mov 0×10(%rbp),%rsi
0×00007fff8a31e711 <dyld_stub_binder_+8>: callq 0×7fff8a31e86d <_Z21_dyld_fast_stub_entryPvl>
0×00007fff8a31e716 <dyld_stub_binder_+13>: mov %rax,%r11
0×00007fff8a31e719 <dyld_stub_binder_+16>: movdqa 0×40(%rsp),%xmm0
0×00007fff8a31e71f <dyld_stub_binder_+22>: movdqa 0×50(%rsp),%xmm1
0×00007fff8a31e725 <dyld_stub_binder_+28>: movdqa 0×60(%rsp),%xmm2
0×00007fff8a31e72b <dyld_stub_binder_+34>: movdqa 0×70(%rsp),%xmm3
0×00007fff8a31e731 <dyld_stub_binder_+40>: movdqa 0×80(%rsp),%xmm4
0×00007fff8a31e73a <dyld_stub_binder_+49>: movdqa 0×90(%rsp),%xmm5
0×00007fff8a31e743 <dyld_stub_binder_+58>: movdqa 0xa0(%rsp),%xmm6
0×00007fff8a31e74c <dyld_stub_binder_+67>: movdqa 0xb0(%rsp),%xmm7
0×00007fff8a31e755 <dyld_stub_binder_+76>: mov (%rsp),%rdi
0×00007fff8a31e759 <dyld_stub_binder_+80>: mov 0×8(%rsp),%rsi
0×00007fff8a31e75e <dyld_stub_binder_+85>: mov 0×10(%rsp),%rdx
0×00007fff8a31e763 <dyld_stub_binder_+90>: mov 0×18(%rsp),%rcx
0×00007fff8a31e768 <dyld_stub_binder_+95>: mov 0×20(%rsp),%r8
0×00007fff8a31e76d <dyld_stub_binder_+100>: mov 0×28(%rsp),%r9
0×00007fff8a31e772 <dyld_stub_binder_+105>: mov 0×30(%rsp),%rax
0×00007fff8a31e777 <dyld_stub_binder_+110>: add $0xc0,%rsp
0×00007fff8a31e77e <dyld_stub_binder_+117>: pop %rbp
0×00007fff8a31e77f <dyld_stub_binder_+118>: add $0×10,%rsp
0×00007fff8a31e783 <dyld_stub_binder_+122>: jmpq *%r11

(gdb) x/2i 0×7fff8fab9a9c
0×7fff8fab9a9c <abort+177>: mov $0×2710,%edi
0×7fff8fab9aa1 <abort+182>: callq 0×7fff8fab9c43 <usleep$nocancel>

(gdb) disass 0×7fff8fab9a9c-5 0×7fff8fab9a9c
Dump of assembler code from 0×7fff8fab9a97 to 0×7fff8fab9a9c:
0×00007fff8fab9a97 <abort+172>: callq 0×7fff8fb1f54a <dyld_stub_kill>
End of assembler dump.

(gdb) disass 0×7fff5fc01000
Dump of assembler code for function __dyld_stub_binding_helper:
0×00007fff5fc01000 <__dyld_stub_binding_helper+0>: add %al,(%rax)
0×00007fff5fc01002 <__dyld_stub_binding_helper+2>: add %al,(%rax)
0×00007fff5fc01004 <__dyld_stub_binding_helper+4>: add %al,(%rax)
0×00007fff5fc01006 <__dyld_stub_binding_helper+6>: add %al,(%rax)
End of assembler dump.

(gdb) x/10 0×7fff5fc01000-0×10
0×7fff5fc00ff0: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01000 <__dyld_stub_binding_helper>: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01010 <__dyld_offset_to_dyld_all_image_infos>: 0×00000000 0×00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 176)

Wednesday, June 6th, 2012

Sometimes, when an application is sluggish, periodically consumes CPU, it is possible to create a set of consecutive memory dumps of the same process to see the temporal development of any thread CPU consumption and figure out potential Spike Interval(s). For example, the following diagram was plotted from !runaway WinDbg command output for thread #1:

The 3rd and the 5th user process memory dumps in addition to increased CPU consumption also have corresponding non-waiting stack trace frames caught while executing some CPU instructions in ModuleA (not preempted with saved context). The first memory dump (yellow bar) with 437 ms user time spent out of 629 ms elapsed time also has a non-waiting stack trace but we consider it a normal application startup CPU consumption spike.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Notes on Memoidealism (1.14)

Wednesday, June 6th, 2012

Here we map objections (and further development) of Berkeley to Locke’s epistemology outlined in the previous note to memoidealistic epistemology:

1. There is no need for special substance to contain primary qualities.

Memory is of the same substance as memories.

2. Ideas exist independently of us in God.

Ideas not contained in memories of specific individuals can be contained either in memories of other individuals or outside memories in Memory.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 60, Mac OS X)

Tuesday, June 5th, 2012

This is a Mac OS X / GDB counterpart to Execution Residue pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8616e82a in __kill ()
#1 0x00007fff8fab9a9c in abort ()
#2 0x000000010269dc29 in bar_5 ()
#3 0x000000010269dc39 in bar_4 ()
#4 0x000000010269dc49 in bar_3 ()
#5 0x000000010269dc59 in bar_2 ()
#6 0x000000010269dc69 in bar_1 ()
#7 0x000000010269dc79 in bar ()
#8 0x000000010269dca0 in main (argc=1, argv=0x7fff6229cb00)

(gdb) x $rsp
0x7fff6229ca38: 0x8fab9a9c

(gdb) x/1000a 0x7fff6229c000
0×7fff6229c000: 0×7fff8947b000 0×7fff8947b570
0×7fff6229c010: 0×4f3ee10c 0×7fff90cb0000
0×7fff6229c020: 0×7fff90cb04d0 0×4e938b16
[…]
0×7fff6229c5f0: 0×7fff622d8d80 0×10269d640
0×7fff6229c600: 0×7fff6229cad0 0×7fff622a460b
0×7fff6229c610: 0×100000000 0×269d000
0×7fff6229c620: 0×7fff6229c630 0×10269db59 <foo_8+9>
0×7fff6229c630: 0×7fff6229c640 0×10269db69 <foo_7+9>
0×7fff6229c640: 0×7fff6229c650 0×10269db79 <foo_6+9>
0×7fff6229c650: 0×7fff6229c660 0×10269db89 <foo_5+9>
0×7fff6229c660: 0×7fff6229c670 0×10269db99 <foo_4+9>
0×7fff6229c670: 0×7fff6229c680 0×10269dba9 <foo_3+9>
0×7fff6229c680: 0×7fff6229c690 0×10269dbb9 <foo_2+9>
0×7fff6229c690: 0×7fff6229c6a0 0×10269dbc9 <foo_1+9>
0×7fff6229c6a0: 0×7fff6229cac0 0×10269dbee <foo+30>

0×7fff6229c6b0: 0×0 0×0
0×7fff6229c6c0: 0×0 0×0
0×7fff6229c6d0: 0×0 0×0
0×7fff6229c6e0: 0×0 0×0
[…]
0×7fff6229c8d0: 0×7fff6229c960 0×7fff622b49cd
0×7fff6229c8e0: 0×10269f05c 0×0
0×7fff6229c8f0: 0×7fff622c465c 0×7fff8a31e5c0 <_Z21dyldGlobalLockReleasev>
0×7fff6229c900: 0×7fff8fab99eb <abort> 0×10269f05c
0×7fff6229c910: 0×101000000000000 0×7fff622d2110
0×7fff6229c920: 0×7fff622d8d80 0×10269f078
0×7fff6229c930: 0×7fff622daac8 0×18
0×7fff6229c940: 0×0 0×0
0×7fff6229c950: 0×10269e030 0×0
0×7fff6229c960: 0×7fff6229c980 0×7fff622a1922
0×7fff6229c970: 0×0 0×0
0×7fff6229c980: 0×7fff6229ca50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6229c990: 0×1 0×7fff6229cb00
0×7fff6229c9a0: 0×7fff6229cb10 0xe223ea612ddc10b7
0×7fff6229c9b0: 0×8 0×0
0×7fff6229c9c0: 0xe223ea612ddc10b7 0×0
0×7fff6229c9d0: 0×0 0×0
0×7fff6229c9e0: 0×585f5f00474e414c 0×20435058005f4350
0×7fff6229c9f0: 0×0 0×0
0×7fff6229ca00: 0×0 0×0
0×7fff6229ca10: 0×0 0×0
0×7fff6229ca20: 0×0 0×0
0×7fff6229ca30: 0×7fff6229ca60 0×7fff8fab9a9c <abort+177>
0×7fff6229ca40: 0×0 0×0
0×7fff6229ca50: 0×7fffffffffdf 0×0
0×7fff6229ca60: 0×7fff6229ca70 0×10269dc29 <bar_5+9>
0×7fff6229ca70: 0×7fff6229ca80 0×10269dc39 <bar_4+9>
0×7fff6229ca80: 0×7fff6229ca90 0×10269dc49 <bar_3+9>
0×7fff6229ca90: 0×7fff6229caa0 0×10269dc59 <bar_2+9>
0×7fff6229caa0: 0×7fff6229cab0 0×10269dc69 <bar_1+9>
0×7fff6229cab0: 0×7fff6229cac0 0×10269dc79 <bar+9>
0×7fff6229cac0: 0×7fff6229cae0 0×10269dca0 <main+32>
0×7fff6229cad0: 0×7fff6229cb00 0×1
0×7fff6229cae0: 0×7fff6229caf0 0×10269db34 <start+52>

0×7fff6229caf0: 0×0 0×1
0×7fff6229cb00: 0×7fff6229cc48 0×0
0×7fff6229cb10: 0×7fff6229ccae 0×7fff6229ccca
[…]

Here’s the source code of the modeling application:

#define def_call(name,x,y) void name##_##x() { name##_##y(); }

#define def_final(name,x) void name##_##x() { }

#define def_final_abort(name,x) void name##_##x() { abort(); }

#define def_init(name,y) void name() { name##_##y(); }

#define def_init_alloc(name,y,size) void name() { int arr[size]; name##_##y(); *arr=0; }

def_final(foo,9)

def_call(foo,8,9)

def_call(foo,7,8)

def_call(foo,6,7)

def_call(foo,5,6)

def_call(foo,4,5)

def_call(foo,3,4)

def_call(foo,2,3)

def_call(foo,1,2)

def_init_alloc(foo,1,256)

def_final_abort(bar,5)

def_call(bar,4,5)

def_call(bar,3,4)

def_call(bar,2,3)

def_call(bar,1,2)

def_init(bar,1)

int main(int argc, const char * argv[])

{

    foo();

    bar();

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

7 Habits of Highly Effective Diagnosticians (Part 0)

Tuesday, June 5th, 2012

Motivated by 7 Habits of Highly Effective Debuggers I would like to reflect on a distinction between diagnostics and problem solving as separate processes (although highly related). First, we reverse the precept from that article because stories such as software logs and traces are of primary importance to software diagnostics (and not only). And without diagnostics there is no effective debugging (treatment, problem solving, etc.)

The Principle Precept of Diagnostics

Stories NOT Statistics secure certainty.

Next parts will be about actual habits so please stay tuned. I would try to finish this list before the forthcoming Webinar on software diagnostics.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 50)

Sunday, June 3rd, 2012

In some cases it is useful to consider Message Context: a set of surrounding messages having some relation to the chosen message:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 49)

Sunday, June 3rd, 2012

Sometimes we need to pay attention to Error Distribution, for example, the distribution of the same error across a software log space or different error messages in different parts of the same software log or trace (providing effective partition):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 48)

Saturday, June 2nd, 2012

If Break-in Activity is usually unrelated to a thread or an adjoint thread which has a discontinuity then Resume Activity pattern highlights messages from that thread:

The difference can be seen on the following graphical representation of two traces we analyzed where in a working trace a break-in preceded resume activity whereas in a non-working trace both patterns were absent:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -