poo - a function that follows foo and bar with a purpose to trigger a crash event, a breakpoint or save memory state.
Examples: void main() { foo(); } void foo() { poo(); } void poo() { asm int 3; }
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
STaMPs - Software Trace and Memory Patterns. Stack Trace and Memory Patterns.
Examples: Got a few visible stamps on this trace. And more stamps on that crash dump.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is a second Webinar from Memory Dump Analysis Services on software diagnostics. The first one is about pattern recognition. During this Webinar you will learn how to apply systems theory and systems thinking for effective and efficient abnormal software behavior diagnostics: the foundation of software troubleshooting and debugging. The seminar summarizes 6 years of research done by Software Diagnostics Institute started with a short blog post Dumps and Systems Theory.
Title: Introduction to Systemic Software Diagnostics: Systems Thinking in Memory Dump and Software Trace Analysis
Date: 3rd of September, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes
Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/377382766
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Memory Dump Analysis Services organizes a free webinar on a unified malware and victimware analysis by using behavioral and structural patterns including a live memory dump analysis example.
Date: 2nd of July, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes
Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/332458406
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Counter Value pattern covers performance monitoring and its logs. A counter value is some variable in memory, for example, a module variable, that is updated periodically to reflect some aspect of state or it can be calculated from different such variables and presented in trace messages. Such messages can also be organized in a similar format as ETW based traces we usually consider as examples for our trace patterns:
Source PID TID Function Value
=================================================
[…]
System 0 0 Committed Memory 12,002,234,654
Process 844 0 Private Bytes 345,206,456
System 0 0 Committed Memory 12,002,236,654
Process 844 0 Working Set 122,160,068
[…]
Therefore, all other trace patterns such as adjoint thread (can be visualized via different colors on a graph), focus of tracing, characteristic message block (for graphs), activity region, significant event, and others can be applicable here. There are also some specific patterns such as global monotonicity and constant value that we discuss with examples in subsequent parts.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Memory Dump Analysis Services will be administering certifications developed by Software Diagnostics Institute for memory dump and software trace analysis:
Software Diagnostics Maturity Enterprise Certification
Memory Dump Analysis Certification is available this September
Debugging TV Frames episode 0×10 contains some background information.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
DumpAnalysis.org portal has been reorganized to Software Diagnostics Institute to reflect the nature of its research activities. More updates later on.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
A NoSQL Problem - when nothing appears on a refresh.
Examples: I got a NoSQL problem when I signed in to that social website.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
While preparing a seminar on Software Diagnostics I made a lot of notes and realized that a system of patterns, corresponding vocabulary and pattern language are needed for this discipline. Here patterns are supposed to be broad in nature and be different from patterns for specific artifacts such as memory dumps and software traces. So the first pattern addresses a diagnostic encounter with a First Fault in comparison to subsequent faults where the problem becomes noticeable and diagnostic resources are allocated. Such faults should not be dismissed. Dan Skwire is a passionate advocate of first fault software problem solving and wrote a book:
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
The following paper proposes distributed control flow reconstruction for first fault diagnosis:
TraceBack: First Fault Diagnosis by Reconstruction of Distributed Control Flow
Memory Dump Analysis Services uses patterns of abnormal software behavior for its first fault diagnostics that doesn’t require any special instrumentation:
Join Debugging Diagnostics Revolution!
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Stack Trace Change is an important pattern for differential memory dump analysis, for example, when memory dumps were generated before and after a problem such as a CPU spike or hang. In the example below we have a normal expected thread stack trace from a memory dump saved before an application was reported unresponsive and another different thread stack trace after:
3 Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen
ChildEBP RetAddr
037dfadc 75210bdd ntdll!ZwWaitForMultipleObjects+0x15
037dfb78 75791a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
037dfbc0 7511086a kernel32!WaitForMultipleObjectsExImplementation+0xe0
037dfc14 00d17c1d user32!RealMsgWaitForMultipleObjectsEx+0x14d
037dfc3c 00ce161d ApplicationA!MsgWaitForMultipleObjects+0x2d
037dfc60 00cdc757 ApplicationA!WaitForSignal+0x1d
037dfc80 00cdaaf6 ApplicationA!WorkLoop+0x57
037dfca4 7579339a ApplicationA!ThreadStart+0x26
037dfcb0 77699ef2 kernel32!BaseThreadInitThunk+0xe
037dfcf0 77699ec5 ntdll!__RtlUserThreadStart+0x70
037dfd08 00000000 ntdll!_RtlUserThreadStart+0x1b
3 Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen
ChildEBP RetAddr
037df38c 752131bb ntdll!ZwDelayExecution+0x15
037df3f4 75213a8b KERNELBASE!SleepEx+0x65
037df404 00d1670b KERNELBASE!Sleep+0xf
037df40c 00d350ef ApplicationA!Sleep+0xb
037df430 6a868aab ApplicationA!PutData+0xbf
037df444 6a8662ec ModuleA!OutputData+0x1b
037df464 00d351de ModuleA!ProcessData+0x16c
037df4a4 00ca8cb4 ApplicationA!SendData+0xbe
[...]
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is a Mac OS X / GDB counterpart to Coincidental Symbolic Information pattern previously described for Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:
(gdb) x $rsp
0x7fff6a162a38: 0x8fab9a9c
(gdb) x/1000a 0x7fff6a162000
[...]
0x7fff6a162960: 0x7fff6a162980 0x7fff6a167922
0x7fff6a162970: 0x0 0x0
0x7fff6a162980: 0x7fff6a162a50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6a162990: 0×1 0×7fff6a162b00
0×7fff6a1629a0: 0×7fff6a162b10 0×7fff6a162bc0
0×7fff6a1629b0: 0×8 0×0
[…]
0×7fff6a162a00: 0×0 0×0
0×7fff6a162a10: 0×0 0×0
0×7fff6a162a20: 0×0 0×0
0×7fff6a162a30: 0×7fff6a162a60 0×7fff8fab9a9c <abort+177>
0×7fff6a162a40: 0×0 0×0
0×7fff6a162a50: 0×7fffffffffdf 0×0
[…]
0×7fff6a163040: 0×35000 0×0
0×7fff6a163050: 0×35000 0×500000007
0×7fff6a163060: 0×7 0×747865745f5f
0×7fff6a163070: 0×0 0×545845545f5f
0×7fff6a163080: 0×0 0×7fff5fc01000 <__dyld_stub_binding_helper>
0×7fff6a163090: 0×22c9d 0xc00001000
0×7fff6a1630a0: 0×0 0×80000400
[…]
(gdb) disass 0×7fff8a31e716
Dump of assembler code for function dyld_stub_binder_:
0×00007fff8a31e709 <dyld_stub_binder_+0>: mov 0×8(%rbp),%rdi
0×00007fff8a31e70d <dyld_stub_binder_+4>: mov 0×10(%rbp),%rsi
0×00007fff8a31e711 <dyld_stub_binder_+8>: callq 0×7fff8a31e86d <_Z21_dyld_fast_stub_entryPvl>
0×00007fff8a31e716 <dyld_stub_binder_+13>: mov %rax,%r11
0×00007fff8a31e719 <dyld_stub_binder_+16>: movdqa 0×40(%rsp),%xmm0
0×00007fff8a31e71f <dyld_stub_binder_+22>: movdqa 0×50(%rsp),%xmm1
0×00007fff8a31e725 <dyld_stub_binder_+28>: movdqa 0×60(%rsp),%xmm2
0×00007fff8a31e72b <dyld_stub_binder_+34>: movdqa 0×70(%rsp),%xmm3
0×00007fff8a31e731 <dyld_stub_binder_+40>: movdqa 0×80(%rsp),%xmm4
0×00007fff8a31e73a <dyld_stub_binder_+49>: movdqa 0×90(%rsp),%xmm5
0×00007fff8a31e743 <dyld_stub_binder_+58>: movdqa 0xa0(%rsp),%xmm6
0×00007fff8a31e74c <dyld_stub_binder_+67>: movdqa 0xb0(%rsp),%xmm7
0×00007fff8a31e755 <dyld_stub_binder_+76>: mov (%rsp),%rdi
0×00007fff8a31e759 <dyld_stub_binder_+80>: mov 0×8(%rsp),%rsi
0×00007fff8a31e75e <dyld_stub_binder_+85>: mov 0×10(%rsp),%rdx
0×00007fff8a31e763 <dyld_stub_binder_+90>: mov 0×18(%rsp),%rcx
0×00007fff8a31e768 <dyld_stub_binder_+95>: mov 0×20(%rsp),%r8
0×00007fff8a31e76d <dyld_stub_binder_+100>: mov 0×28(%rsp),%r9
0×00007fff8a31e772 <dyld_stub_binder_+105>: mov 0×30(%rsp),%rax
0×00007fff8a31e777 <dyld_stub_binder_+110>: add $0xc0,%rsp
0×00007fff8a31e77e <dyld_stub_binder_+117>: pop %rbp
0×00007fff8a31e77f <dyld_stub_binder_+118>: add $0×10,%rsp
0×00007fff8a31e783 <dyld_stub_binder_+122>: jmpq *%r11
(gdb) x/2i 0×7fff8fab9a9c
0×7fff8fab9a9c <abort+177>: mov $0×2710,%edi
0×7fff8fab9aa1 <abort+182>: callq 0×7fff8fab9c43 <usleep$nocancel>
(gdb) disass 0×7fff8fab9a9c-5 0×7fff8fab9a9c
Dump of assembler code from 0×7fff8fab9a97 to 0×7fff8fab9a9c:
0×00007fff8fab9a97 <abort+172>: callq 0×7fff8fb1f54a <dyld_stub_kill>
End of assembler dump.
(gdb) disass 0×7fff5fc01000
Dump of assembler code for function __dyld_stub_binding_helper:
0×00007fff5fc01000 <__dyld_stub_binding_helper+0>: add %al,(%rax)
0×00007fff5fc01002 <__dyld_stub_binding_helper+2>: add %al,(%rax)
0×00007fff5fc01004 <__dyld_stub_binding_helper+4>: add %al,(%rax)
0×00007fff5fc01006 <__dyld_stub_binding_helper+6>: add %al,(%rax)
End of assembler dump.
(gdb) x/10 0×7fff5fc01000-0×10
0×7fff5fc00ff0: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01000 <__dyld_stub_binding_helper>: 0×00000000 0×00000000 0×00000000 0×00000000
0×7fff5fc01010 <__dyld_offset_to_dyld_all_image_infos>: 0×00000000 0×00000000
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Forthcoming Training: Accelerated Mac OS X Core Dump Analysis
Sometimes, when an application is sluggish, periodically consumes CPU, it is possible to create a set of consecutive memory dumps of the same process to see the temporal development of any thread CPU consumption and figure out potential Spike Interval(s). For example, the following diagram was plotted from !runaway WinDbg command output for thread #1:
The 3rd and the 5th user process memory dumps in addition to increased CPU consumption also have corresponding non-waiting stack trace frames caught while executing some CPU instructions in ModuleA (not preempted with saved context). The first memory dump (yellow bar) with 437 ms user time spent out of 629 ms elapsed time also has a non-waiting stack trace but we consider it a normal application startup CPU consumption spike.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Here we map objections (and further development) of Berkeley to Locke’s epistemology outlined in the previous note to memoidealistic epistemology:
1. There is no need for special substance to contain primary qualities.
Memory is of the same substance as memories.
2. Ideas exist independently of us in God.
Ideas not contained in memories of specific individuals can be contained either in memories of other individuals or outside memories in Memory.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is a Mac OS X / GDB counterpart to Execution Residue pattern previously described for Windows platforms:
(gdb) bt
#0 0x00007fff8616e82a in __kill ()
#1 0x00007fff8fab9a9c in abort ()
#2 0x000000010269dc29 in bar_5 ()
#3 0x000000010269dc39 in bar_4 ()
#4 0x000000010269dc49 in bar_3 ()
#5 0x000000010269dc59 in bar_2 ()
#6 0x000000010269dc69 in bar_1 ()
#7 0x000000010269dc79 in bar ()
#8 0x000000010269dca0 in main (argc=1, argv=0x7fff6229cb00)
(gdb) x $rsp
0x7fff6229ca38: 0x8fab9a9c
(gdb) x/1000a 0x7fff6229c000
0×7fff6229c000: 0×7fff8947b000 0×7fff8947b570
0×7fff6229c010: 0×4f3ee10c 0×7fff90cb0000
0×7fff6229c020: 0×7fff90cb04d0 0×4e938b16
[…]
0×7fff6229c5f0: 0×7fff622d8d80 0×10269d640
0×7fff6229c600: 0×7fff6229cad0 0×7fff622a460b
0×7fff6229c610: 0×100000000 0×269d000
0×7fff6229c620: 0×7fff6229c630 0×10269db59 <foo_8+9>
0×7fff6229c630: 0×7fff6229c640 0×10269db69 <foo_7+9>
0×7fff6229c640: 0×7fff6229c650 0×10269db79 <foo_6+9>
0×7fff6229c650: 0×7fff6229c660 0×10269db89 <foo_5+9>
0×7fff6229c660: 0×7fff6229c670 0×10269db99 <foo_4+9>
0×7fff6229c670: 0×7fff6229c680 0×10269dba9 <foo_3+9>
0×7fff6229c680: 0×7fff6229c690 0×10269dbb9 <foo_2+9>
0×7fff6229c690: 0×7fff6229c6a0 0×10269dbc9 <foo_1+9>
0×7fff6229c6a0: 0×7fff6229cac0 0×10269dbee <foo+30>
0×7fff6229c6b0: 0×0 0×0
0×7fff6229c6c0: 0×0 0×0
0×7fff6229c6d0: 0×0 0×0
0×7fff6229c6e0: 0×0 0×0
[…]
0×7fff6229c8d0: 0×7fff6229c960 0×7fff622b49cd
0×7fff6229c8e0: 0×10269f05c 0×0
0×7fff6229c8f0: 0×7fff622c465c 0×7fff8a31e5c0 <_Z21dyldGlobalLockReleasev>
0×7fff6229c900: 0×7fff8fab99eb <abort> 0×10269f05c
0×7fff6229c910: 0×101000000000000 0×7fff622d2110
0×7fff6229c920: 0×7fff622d8d80 0×10269f078
0×7fff6229c930: 0×7fff622daac8 0×18
0×7fff6229c940: 0×0 0×0
0×7fff6229c950: 0×10269e030 0×0
0×7fff6229c960: 0×7fff6229c980 0×7fff622a1922
0×7fff6229c970: 0×0 0×0
0×7fff6229c980: 0×7fff6229ca50 0×7fff8a31e716 <dyld_stub_binder_+13>
0×7fff6229c990: 0×1 0×7fff6229cb00
0×7fff6229c9a0: 0×7fff6229cb10 0xe223ea612ddc10b7
0×7fff6229c9b0: 0×8 0×0
0×7fff6229c9c0: 0xe223ea612ddc10b7 0×0
0×7fff6229c9d0: 0×0 0×0
0×7fff6229c9e0: 0×585f5f00474e414c 0×20435058005f4350
0×7fff6229c9f0: 0×0 0×0
0×7fff6229ca00: 0×0 0×0
0×7fff6229ca10: 0×0 0×0
0×7fff6229ca20: 0×0 0×0
0×7fff6229ca30: 0×7fff6229ca60 0×7fff8fab9a9c <abort+177>
0×7fff6229ca40: 0×0 0×0
0×7fff6229ca50: 0×7fffffffffdf 0×0
0×7fff6229ca60: 0×7fff6229ca70 0×10269dc29 <bar_5+9>
0×7fff6229ca70: 0×7fff6229ca80 0×10269dc39 <bar_4+9>
0×7fff6229ca80: 0×7fff6229ca90 0×10269dc49 <bar_3+9>
0×7fff6229ca90: 0×7fff6229caa0 0×10269dc59 <bar_2+9>
0×7fff6229caa0: 0×7fff6229cab0 0×10269dc69 <bar_1+9>
0×7fff6229cab0: 0×7fff6229cac0 0×10269dc79 <bar+9>
0×7fff6229cac0: 0×7fff6229cae0 0×10269dca0 <main+32>
0×7fff6229cad0: 0×7fff6229cb00 0×1
0×7fff6229cae0: 0×7fff6229caf0 0×10269db34 <start+52>
0×7fff6229caf0: 0×0 0×1
0×7fff6229cb00: 0×7fff6229cc48 0×0
0×7fff6229cb10: 0×7fff6229ccae 0×7fff6229ccca
[…]
Here’s the source code of the modeling application:
#define def_call(name,x,y) void name##_##x() { name##_##y(); }
#define def_final(name,x) void name##_##x() { }
#define def_final_abort(name,x) void name##_##x() { abort(); }
#define def_init(name,y) void name() { name##_##y(); }
#define def_init_alloc(name,y,size) void name() { int arr[size]; name##_##y(); *arr=0; }
def_final(foo,9)
def_call(foo,8,9)
def_call(foo,7,8)
def_call(foo,6,7)
def_call(foo,5,6)
def_call(foo,4,5)
def_call(foo,3,4)
def_call(foo,2,3)
def_call(foo,1,2)
def_init_alloc(foo,1,256)
def_final_abort(bar,5)
def_call(bar,4,5)
def_call(bar,3,4)
def_call(bar,2,3)
def_call(bar,1,2)
def_init(bar,1)
int main(int argc, const char * argv[])
{
foo();
bar();
}
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Forthcoming Training: Accelerated Mac OS X Core Dump Analysis
Motivated by 7 Habits of Highly Effective Debuggers I would like to reflect on a distinction between diagnostics and problem solving as separate processes (although highly related). First, we reverse the precept from that article because stories such as software logs and traces are of primary importance to software diagnostics (and not only). And without diagnostics there is no effective debugging (treatment, problem solving, etc.)
The Principle Precept of Diagnostics
Stories NOT Statistics secure certainty.
Next parts will be about actual habits so please stay tuned. I would try to finish this list before the forthcoming Webinar on software diagnostics.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In some cases it is useful to consider Message Context: a set of surrounding messages having some relation to the chosen message:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we need to pay attention to Error Distribution, for example, the distribution of the same error across a software log space or different error messages in different parts of the same software log or trace (providing effective partition):
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
If Break-in Activity is usually unrelated to a thread or an adjoint thread which has a discontinuity then Resume Activity pattern highlights messages from that thread:
The difference can be seen on the following graphical representation of two traces we analyzed where in a working trace a break-in preceded resume activity whereas in a non-working trace both patterns were absent:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We resume our software trace analysis pattern catalog. The next pattern is called Break-in Activity. This is a message or a set of messages that surface just before the end of discontinuity of a adjoint thread and possibly triggered it:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Was browsing Amazon and found this book with a cover like an artificially colored memory image and text in black gaps but rotated by 90 degrees counterclockwise:
Noise Channels: Glitch and Error in Digital Culture (Electronic Mediations)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
