Displaying thread stack trace is the most used action in crash or core dump analysis and debugging. To show various available GDB commands I created the next version of the test program with the following source code:
#include <stdio.h>
void func_1(int param_1, char param_2, int *param_3, char *param_4);
void func_2(int param_1, char param_2, int *param_3, char *param_4);
void func_3(int param_1, char param_2, int *param_3, char *param_4);
void func_4();
int val_1;
char val_2;
int *pval_1 = &val_1;
char *pval_2 = &val_2;
int main()
{
val_1 = 1;
val_2 = '1';
func_1(val_1, val_2, (int *)pval_1, (char *)pval_2);
return 0;
}
void func_1(int param_1, char param_2, int *param_3, char *param_4)
{
val_1 = 2;
val_2 = '2';
func_2(param_1, param_2, param_3, param_4);
}
void func_2(int param_1, char param_2, int *param_3, char *param_4)
{
val_1 = 3;
val_2 = '3';
func_3(param_1, param_2, param_3, param_4);
}
void func_3(int param_1, char param_2, int *param_3, char *param_4)
{
*pval_1 += param_1;
*pval_2 += param_2;
func_4();
}
void func_4()
{
puts("Hello World!");
}
I compiled it with -g gcc compiler option to generate symbolic information. It will be needed for GDB to display function arguments and local variables.
C:\MinGW\examples>..\bin\gcc -g -o test.exe test.c
If you have a crash in func_4 then you can examine stack trace (backtrace) once you open a core dump. Because we don’t have a core dump of our test program we will simulate the stack trace by putting a breakpoint on func_4. In GDB this can be done by break command:
C:\MinGW\examples>..\bin\gdb test.exe
...
...
...
(gdb) break func_4
Breakpoint 1 at 0x40141d
(gdb) run
Starting program: C:\MinGW\examples/test.exeBreakpoint 1, 0x0040141d in func_4 ()
(gdb)
In WinDbg the breakpoint command is bp:
CommandLine: C:\dmitri\test\release\test.exe
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 0040f000 test.exe
ModLoad: 7d4c0000 7d5f0000 NOT_AN_IMAGE
ModLoad: 7d600000 7d6f0000 C:\W2K3\SysWOW64\ntdll32.dll
ModLoad: 7d4c0000 7d5f0000 C:\W2K3\syswow64\kernel32.dll
(103c.17d8): Break instruction exception - code 80000003 (first chance)
eax=7d600000 ebx=7efde000 ecx=00000005 edx=00000020 esi=7d6a01f4 edi=00221f38
eip=7d61002d esp=0012fb4c ebp=0012fcac iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll32!DbgBreakPoint:
7d61002d cc int 3
0:000> bp func_4
0:000> g
ModLoad: 71c20000 71c32000 C:\W2K3\SysWOW64\tsappcmp.dll
ModLoad: 77ba0000 77bfa000 C:\W2K3\syswow64\msvcrt.dll
ModLoad: 77f50000 77fec000 C:\W2K3\syswow64\ADVAPI32.dll
ModLoad: 7da20000 7db00000 C:\W2K3\syswow64\RPCRT4.dll
Breakpoint 0 hit
eax=0040c9d0 ebx=7d4d8dc9 ecx=0040c9d0 edx=00000064 esi=00000002 edi=00000ece
eip=00408be0 esp=0012ff24 ebp=0012ff28 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
test!func_4:
00408be0 55 push ebp
I had to disable optimization in the project properties otherwise Visual C++ compiler optimizes away all function calls and produces the following short code:
0:000> uf main
00401000 push offset test!`string' (004020f4)
00401005 mov dword ptr [test!val_1 (0040337c)],4
0040100f mov byte ptr [test!val_2 (00403378)],64h
00401016 call dword ptr [test!_imp__puts (004020a0)]
0040101c add esp,4
0040101f xor eax,eax
00401021 ret
I will talk about setting breakpoints in another part and here I’m going to concentrate only on commands that examine call stack. backtrace or bt command shows stack trace. backtrace <N> or bt <N> shows only the innermost N stack frames. backtrace -<N> or bt -<N> shows only the outermost N stack frames. backtrace full or bt full additionally shows local variables. There are also variants backtrace full <N> or bt full <N> and backtrace full -<N> or bt full -<N>:
(gdb) backtrace
#0 func_4 () at test.c:48
#1 0x00401414 in func_3 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:43
#2 0x004013da in func_2 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:35
#3 0x0040139a in func_1 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:27
#4 0x00401355 in main () at test.c:18
(gdb) bt
#0 func_4 () at test.c:48
#1 0x00401414 in func_3 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:43
#2 0x004013da in func_2 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:35
#3 0x0040139a in func_1 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:27
#4 0x00401355 in main () at test.c:18
(gdb) bt 2
#0 func_4 () at test.c:48
#1 0x00401414 in func_3 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:43
(More stack frames follow...)
(gdb) bt -2
#3 0x0040139a in func_1 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:27
#4 0x00401355 in main () at test.c:18
(gdb) bt full
#0 func_4 () at test.c:48
No locals.
#1 0x00401414 in func_3 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:43
param_2 = 49 '1'
#2 0x004013da in func_2 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:35
param_2 = 49 '1'
#3 0x0040139a in func_1 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:27
param_2 = 49 '1'
#4 0x00401355 in main () at test.c:18
No locals.
(gdb) bt full 2
#0 func_4 () at test.c:48
No locals.
#1 0x00401414 in func_3 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:43
param_2 = 49 '1'
(More stack frames follow...)
(gdb) bt full -2
#3 0x0040139a in func_1 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:27
param_2 = 49 '1'
#4 0x00401355 in main () at test.c:18
No locals.
(gdb)
In WinDbg there is only one k command but it has many parameters, for example:
- default stack trace with source code lines
0:000> k
ChildEBP RetAddr
0012ff20 00408c30 test!func_4 [c:\dmitri\test\test\test.cpp @ 47]
0012ff28 00408c69 test!func_3+0x30 [c:\dmitri\test\test\test.cpp @ 44]
0012ff40 00408c99 test!func_2+0x29 [c:\dmitri\test\test\test.cpp @ 35]
0012ff58 00408cd3 test!func_1+0x29 [c:\dmitri\test\test\test.cpp @ 27]
0012ff70 00401368 test!main+0x33 [c:\dmitri\test\test\test.cpp @ 18]
0012ffc0 7d4e992a test!__tmainCRTStartup+0x15f [f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c @ 327]
0012fff0 00000000 kernel32!BaseProcessStart+0x28
- stack trace without source code lines
0:000> kL
ChildEBP RetAddr
0012ff20 00408c30 test!func_4
0012ff28 00408c69 test!func_3+0x30
0012ff40 00408c99 test!func_2+0x29
0012ff58 00408cd3 test!func_1+0x29
0012ff70 00401368 test!main+0x33
0012ffc0 7d4e992a test!__tmainCRTStartup+0x15f
0012fff0 00000000 kernel32!BaseProcessStart+0x28
- full stack trace without source code lines showing 3 stack arguments for every stack frame, calling convention and optimization information
0:000> kvL
ChildEBP RetAddr Args to Child
0012ff20 00408c30 0012ff40 00408c69 00000001 test!func_4 (CONV: cdecl)
0012ff28 00408c69 00000001 00000031 0040c9d4 test!func_3+0x30 (CONV: cdecl)
0012ff40 00408c99 00000001 00000031 0040c9d4 test!func_2+0x29 (CONV: cdecl)
0012ff58 00408cd3 00000001 00000031 0040c9d4 test!func_1+0x29 (CONV: cdecl)
0012ff70 00401368 00000001 004230e0 00423120 test!main+0x33 (CONV: cdecl)
0012ffc0 7d4e992a 00000000 00000000 7efde000 test!__tmainCRTStartup+0x15f (FPO: [Non-Fpo]) (CONV: cdecl)
0012fff0 00000000 004013bf 00000000 00000000 kernel32!BaseProcessStart+0x28 (FPO: [Non-Fpo])
- stack trace without source code lines showing all function parameters
0:000> kPL
ChildEBP RetAddr
0012ff20 00408c30 test!func_4(void)
0012ff28 00408c69 test!func_3(
int param_1 = 1,
char param_2 = 49 '1',
int * param_3 = 0x0040c9d4,
char * param_4 = 0x0040c9d0 "d")+0x30
0012ff40 00408c99 test!func_2(
int param_1 = 1,
char param_2 = 49 '1',
int * param_3 = 0x0040c9d4,
char * param_4 = 0x0040c9d0 "d")+0x29
0012ff58 00408cd3 test!func_1(
int param_1 = 1,
char param_2 = 49 '1',
int * param_3 = 0x0040c9d4,
char * param_4 = 0x0040c9d0 "d")+0x29
0012ff70 00401368 test!main(void)+0x33
0012ffc0 7d4e992a test!__tmainCRTStartup(void)+0x15f
0012fff0 00000000 kernel32!BaseProcessStart+0x28
- stack trace without source code lines showing stack frame numbers
0:000> knL
# ChildEBP RetAddr
00 0012ff20 00408c30 test!func_4
01 0012ff28 00408c69 test!func_3+0x30
02 0012ff40 00408c99 test!func_2+0x29
03 0012ff58 00408cd3 test!func_1+0x29
04 0012ff70 00401368 test!main+0x33
05 0012ffc0 7d4e992a test!__tmainCRTStartup+0x15f
06 0012fff0 00000000 kernel32!BaseProcessStart+0x28
- stack trace without source code lines showing the distance between stack frames in bytes
0:000> knfL
# Memory ChildEBP RetAddr
00 0012ff20 00408c30 test!func_4
01 8 0012ff28 00408c69 test!func_3+0x30
02 18 0012ff40 00408c99 test!func_2+0x29
03 18 0012ff58 00408cd3 test!func_1+0x29
04 18 0012ff70 00401368 test!main+0x33
05 50 0012ffc0 7d4e992a test!__tmainCRTStartup+0x15f
06 30 0012fff0 00000000 kernel32!BaseProcessStart+0x28
- stack trace without source code lines showing the innermost 2 frames:
0:000> kL 2
ChildEBP RetAddr
0012ff20 00408c30 test!func_4
0012ff28 00408c69 test!func_3+0x30
If you want to see stack traces from all threads in a process use the following command:
(gdb) thread apply all bt
Thread 1 (thread 728.0xc0c):
#0 func_4 () at test.c:48
#1 0x00401414 in func_3 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:43
#2 0x004013da in func_2 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:35
#3 0x0040139a in func_1 (param_1=1, param_2=49 '1', param_3=0x404080,
param_4=0x404070 "d") at test.c:27
#4 0x00401355 in main () at test.c:18
(gdb)
In WinDbg it is ~*k. Any parameter shown above can be used, for example:
0:000> ~*kL
. 0 Id: 103c.17d8 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0012ff20 00408c30 test!func_4
0012ff28 00408c69 test!func_3+0x30
0012ff40 00408c99 test!func_2+0x29
0012ff58 00408cd3 test!func_1+0x29
0012ff70 00401368 test!main+0x33
0012ffc0 7d4e992a test!__tmainCRTStartup+0x15f
0012fff0 00000000 kernel32!BaseProcessStart+0x28
Therefore, our next version of the map contains these new commands:
Action | GDB | WinDbg
----------------------------------------------------------
Start the process | run | g
Exit | (q)uit | q
Disassemble (forward) | (disas)semble | uf, u
Disassemble N instructions | x/<N>i | -
Disassemble (backward) | - | ub
Stack trace | backtrace (bt) | k
Full stack trace | bt full | kv
Partial trace (innermost) | bt <N> | k <N>
Partial trace (outermost) | bt -<N> | -
Stack trace for all threads | thread apply all bt | ~*k
Breakpoint | break | bp
- Dmitry Vostokov @ DumpAnalysis.org -
I’m introducing the new blog category where I’m going to dig into crash dump analysis on Mac OS X and FreeBSD whenever an occasion happens.
to write about my experience:
