Another common mistake I observe is relying on what debuggers report without double-checking. Present day debuggers, like WinDbg or GDB, are symbol-driven, they do not possess much of that semantic knowledge that a human debugger has. Also, it is better to report more than less: what is irrelevant can be skipped over by a skilled memory analyst but what looks suspicious to the problem at hand shall be double-checked to see if it is not coincidental. One example we consider here is Coincidental Symbolic Information.
An application is frequently crashing. The process memory dump file shows only one thread left inside without any exception handling frames. In order to hypothesize about the probable cause the thread raw stack data is analyzed. It shows a few C++ STL calls with a custom smart pointer class and memory allocator like this:
app!std::vector<SmartPtr<ClassA>, std::allocator<SmartPtr<ClassA> > >::operator[]+
WinDbg !analyze-v command output points to this code:
FOLLOWUP_IP:
app!std::bad_alloc::~bad_alloc <PERF> (app+0x0)+0
00400000 4d dec ebp
Raw stack data contains a few symbolic references to bad_alloc destructor too:
[...]
0012f9c0 00000100
0012f9c4 00400100 app!std::bad_alloc::~bad_alloc <PERF> (app+0x100)
0012f9c8 00000000
0012f9cc 0012f9b4
0012f9d0 00484488 app!_NULL_IMPORT_DESCRIPTOR+0x1984
0012f9d4 0012fa8c
0012f9d8 7c828290 ntdll!_except_handler3
0012f9dc 0012fa3c
0012f9e0 7c82b04a ntdll!RtlImageNtHeaderEx+0xee
0012f9e4 00482f08 app!_NULL_IMPORT_DESCRIPTOR+0x404
0012f9e8 00151ed0
0012f9ec 00484c1e app!_NULL_IMPORT_DESCRIPTOR+0x211a
0012f9f0 00000100
0012f9f4 00400100 app!std::bad_alloc::~bad_alloc <PERF> (app+0x100)
[...]
By linking all these three pieces together an engineer hypothesized that the cause of failure is memory allocation. However, careful analysis reveals all of them as a coincidental symbolic information and renders hypothesis much less plausible:
1. The address of app!std::bad_alloc::~bad_alloc is 00400000 which coincides with the start of the main application module:
0:000> lm a 00400000
start end module name
00400000 004c4000 app (no symbols)
As a consequence, its assembly language code makes no sense:
0:000> u 00400000
app:
00400000 4d dec ebp
00400001 5a pop edx
00400002 90 nop
00400003 0003 add byte ptr [ebx],al
00400005 0000 add byte ptr [eax],al
00400007 000400 add byte ptr [eax+eax],al
0040000a 0000 add byte ptr [eax],al
0040000c ff ???
2. All std::vector references are in fact fragments of a UNICODE string that can be dumped using du command:
[...]
0012ef14 00430056 app!std::vector<SmartPtr<ClassA>, std::allocator<SmartPtr<ClassA> > >::operator[]+0x16
0012ef18 00300038
0012ef1c 0043002e app!std::vector<SmartPtr<ClassA>, std::allocator<SmartPtr<ClassA> > >::size+0x1
[...]
0:000> du 0012ef14 l6
0012ef14 "VC80.C"
3. Raw stack data references to bad_alloc destructor are still module addresses in disguise, 00400100 or app+0×100, with nonsense assembly code:
0:000> u 00400100
app+0x100:
00400100 50 push eax
00400101 45 inc ebp
00400102 0000 add byte ptr [eax],al
00400104 4c dec esp
00400105 010500571aac add dword ptr ds:[0AC1A5700h],eax
0040010b 4a dec edx
0040010c 0000 add byte ptr [eax],al
0040010e 0000 add byte ptr [eax],al
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
