This is a small case study to complement Incorrect Stack Trace pattern and show how to reconstruct stack trace manually based on an example with complete source code.
I created a small working multithreaded program:
#include "stdafx.h"
#include <stdio.h>
#include <process.h>
typedef void (*REQ_JUMP)();
typedef void (*REQ_RETURN)();
const char str[] = "\0\0\0\0\0\0\0";
bool loop = true;
void return_func()
{
puts("Return Func");
loop = false;
_endthread();
}
void jump_func()
{
puts("Jump Func");
}
void internal_func_2(void *param_jump,void *param_return)
{
REQ_JUMP f_jmp = (REQ_JUMP)param_jump;
REQ_RETURN f_ret = (REQ_RETURN)param_return;
puts("Internal Func 2");
// Uncomment memcpy to crash the program
// Overwrite f_jmp and f_ret with NULL
// memcpy(&f_ret, str, sizeof(str));
__asm
{
push f_ret;
mov eax, f_jmp
mov ebp, 0 // use ebp as a general purpose register
jmp eax
}
}
void internal_func_1(void *param)
{
puts("Internal Func 1");
internal_func_2(param, &return_func);
}
void thread_request(void *param)
{
puts("Request");
internal_func_1(param);
}
int _tmain(int argc, _TCHAR* argv[])
{
_beginthread(thread_request, 0, (void *)jump_func);
while (loop);
return 0;
}
For it I had to disable optimizations in Visual C++ compiler otherwise most of the code would have been eliminated because the program is very small and easy for code optimizer. If we run the program it displays the following output:
Request
Internal Func 1
Internal Func 2
Jump Func
Return Func
internal_func_2 gets two parameters: the function address to jump and the function address to call upon the return. The latter sets loop variable to false in order to break infinite main thread loop and calls _endthread. Why is that complexity in so small sample? I wanted to simulate FPO optimization in an inner function call and also gain control over a return address. This is why I set EBP to zero before jumping and pushed the custom return address which I can change any time. If I used the call instruction then the processor would have determined the return address as the next instruction address.
The code also copies two internal_func_2 parameters into local variables f_jmp and f_ret because the commented memcpy call is crafted to overwrite them with zeroes and do not touch the saved EBP, return address and function arguments. This is all to make stack trace incorrect but at the same time make manual stack reconstruction as easy as possible in this example.
Let’s suppose that memcpy call is a bug that overwrites local variables. Then we have a crash obviously because EAX is zero and jump to zero address will cause access violation. EBP is also 0 because we assigned 0 to it explicitly. Let’s pretend that we wanted to pass some constant via EBP and it is zero.
What we have now:
EBP is 0
EIP is 0
the return address is 0
As you might have expected already when you load a crash dump WinDbg is utterly confused because it has no clue on how to reconstruct the stack trace:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(bd0.ec8): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=00595620 ecx=00000002 edx=00000000 esi=00000000 edi=00000000
eip=00000000 esp=0069ff54 ebp=00000000 iopl=0 nv up ei pl nz ac po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010212
00000000 ?? ???
0:001> kv
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0069ff50 00000000 00000000 00000000 0069ff70 0×0
Fortunately ESP is not zero so we can look at raw stack:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000
0069ff60 0069ff70
0069ff64 0040187f WrongIP!internal_func_1+0x1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401840 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 0040189c WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
0069ff80 78132848 msvcr80!_endthread+0x4b
0069ff84 00401830 WrongIP!jump_func
0069ff88 aa75565b
0069ff8c 00000000
0069ff90 00000000
0069ff94 00595620
0069ff98 c0000005
0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac d207e277
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0x34
0069ffc0 00595620
0069ffc4 00000000
0069ffc8 00000000
0069ffcc 00595620
0069ffd0 c0000005
Here we can start searching for the following pairs:
EBP: PreviousEBP
Function return address
…
…
…
PreviousEBP: PrePreviousEBP
Function return address
…
…
…
for example:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000
0069ff60 0069ff70
0069ff64 0040187f WrongIP!internal_func_1+0×1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401840 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 0040189c WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
This is based on the fact that a function call saves its return address and the standard function prolog saves the previous EBP value and sets ESP to point to it.
push ebp
mov ebp, esp
Therefore our stack looks like this:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000
0069ff60 0069ff70
0069ff64 0040187f WrongIP!internal_func_1+0×1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401840 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 0040189c WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
0069ff80 78132848 msvcr80!_endthread+0×4b
0069ff84 00401830 WrongIP!jump_func
0069ff88 aa75565b
0069ff8c 00000000
0069ff90 00000000
0069ff94 00595620
0069ff98 c0000005
0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac d207e277
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0×34
0069ffc0 00595620
0069ffc4 00000000
0069ffc8 00000000
0069ffcc 00595620
0069ffd0 c0000005
Also we double check return addresses to see if they are valid code indeed. The best way is to try to disassemble them backwards. This should show call instructions resulted in saved return addresses:
0:001> ub WrongIP!internal_func_1+0x1f
WrongIP!internal_func_1+0x1:
00401871 mov ebp,esp
00401873 push offset WrongIP!GS_ExceptionPointers+0x38 (00402124)
00401878 call dword ptr [WrongIP!_imp__puts (004020ac)]
0040187e add esp,4
00401881 push offset WrongIP!return_func (00401850)
00401886 mov eax,dword ptr [ebp+8]
00401889 push eax
0040188a call WrongIP!internal_func_2 (004017e0)
0:001> ub WrongIP!thread_request+0xc
WrongIP!internal_func_1+0x2d:
0040189d int 3
0040189e int 3
0040189f int 3
WrongIP!thread_request:
004018a0 push ebp
004018a1 mov ebp,esp
004018a3 mov eax,dword ptr [ebp+8]
004018a6 push eax
004018a7 call WrongIP!internal_func_1 (00401870)
0:001> ub msvcr80!_endthread+0x4b
msvcr80!_endthread+0x2f:
7813282c pop esi
7813282d push 0Ch
7813282f push offset msvcr80!__rtc_tzz+0x64 (781b4b98)
78132834 call msvcr80!_SEH_prolog4 (78138c80)
78132839 call msvcr80!_getptd (78132e29)
7813283e and dword ptr [ebp-4],0
78132842 push dword ptr [eax+58h]
78132845 call dword ptr [eax+54h]
0:001> ub msvcr80!_endthread+0xcb
msvcr80!_endthread+0xaf:
781328ac mov edx,dword ptr [ecx+58h]
781328af mov dword ptr [eax+58h],edx
781328b2 mov edx,dword ptr [ecx+4]
781328b5 push ecx
781328b6 mov dword ptr [eax+4],edx
781328b9 call msvcr80!_freefls (78132e41)
781328be call msvcr80!_initp_misc_winxfltr (781493c1)
781328c3 call msvcr80!_endthread+0×30 (7813282d)
0:001> ub BaseThreadStart+0x34
kernel32!BaseThreadStart+0x10:
7d4dfdfd mov eax,dword ptr fs:[00000018h]
7d4dfe03 cmp dword ptr [eax+10h],1E00h
7d4dfe0a jne kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe0c cmp byte ptr [kernel32!BaseRunningInServerProcess (7d560008)],0
7d4dfe13 jne kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe15 call dword ptr [kernel32!_imp__CsrNewThread (7d4d0310)]
7d4dfe1b push dword ptr [ebp+0Ch]
7d4dfe1e call dword ptr [ebp+8]
Now we can use extended version of k command and supply custom EBP, ESP and EIP values. We set EBP to the first found address of EBP:PreviousEBP pair and set EIP to 0:
0:001> k L=0069ff60 0069ff60 0
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0069ff5c 0069ff70 0x0
0069ff60 0040188f 0x69ff70
0069ff70 004018ac WrongIP!internal_func_1+0x1f
0069ff7c 78132848 WrongIP!thread_request+0xc
0069ffb4 781328c8 msvcr80!_endthread+0x4b
0069ffb8 7d4dfe21 msvcr80!_endthread+0xcb
0069ffec 00000000 kernel32!BaseThreadStart+0x34
The stack trace looks good because it also shows BaseThreadStart.
From the backwards disassembly of the return address WrongIP!internal_func_1+0×1f we see that internal_func_1 calls internal_func_2 so we can disassemble the latter function:
0:001> uf internal_func_2
Flow analysis was incomplete, some code may be missing
WrongIP!internal_func_2:
28 004017e0 push ebp
28 004017e1 mov ebp,esp
28 004017e3 sub esp,8
29 004017e6 mov eax,dword ptr [ebp+8]
29 004017e9 mov dword ptr [ebp-4],eax
30 004017ec mov ecx,dword ptr [ebp+0Ch]
30 004017ef mov dword ptr [ebp-8],ecx
32 004017f2 push offset WrongIP!GS_ExceptionPointers+0×28 (00402114)
32 004017f7 call dword ptr [WrongIP!_imp__puts (004020ac)]
32 004017fd add esp,4
33 00401800 push 8
33 00401802 push offset WrongIP!GS_ExceptionPointers+0×8 (004020f4)
33 00401807 lea edx,[ebp-8]
33 0040180a push edx
33 0040180b call WrongIP!memcpy (00401010)
33 00401810 add esp,0Ch
35 00401813 push dword ptr [ebp-8]
36 00401816 mov eax,dword ptr [ebp-4]
37 00401819 mov ebp,0
38 0040181e jmp eax
We see that it takes some value from [ebp-8], puts it into EAX and then jumps to that address. The function uses standard prolog (in blue) and therefore EBP-4 is the local variable. From the code we see that it comes from [EBP+8] which is the first function parameter:
EBP+C: second parameter
EBP+8: first parameter
EBP+4: return address
EBP: previous EBP
EBP-4: local variable
EBP-8: local variable
If we examine the first parameter we would see it is a valid function address that we were supposed to call:
0:001> kv L=0069ff60 0069ff60 0
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0069ff5c 0069ff70 0040188f 00401830 00401850 0x0
0069ff60 0040188f 00401830 00401850 0069ff7c 0x69ff70
0069ff70 004018ac 00401830 0069ffb4 78132848 WrongIP!internal_func_1+0×1f
0069ff7c 78132848 00401830 6d5ba283 00000000 WrongIP!thread_request+0xc
0069ffb4 781328c8 7d4dfe21 00595620 00000000 msvcr80!_endthread+0×4b
0069ffb8 7d4dfe21 00595620 00000000 00000000 msvcr80!_endthread+0xcb
0069ffec 00000000 7813286e 00595620 00000000 kernel32!BaseThreadStart+0×34
0:001> u 00401830
WrongIP!jump_func:
00401830 push ebp
00401831 mov ebp,esp
00401833 push offset WrongIP!GS_ExceptionPointers+0x1c (00402108)
00401838 call dword ptr [WrongIP!_imp__puts (004020ac)]
0040183e add esp,4
00401841 pop ebp
00401842 ret
00401843 int 3
However if we look at the code we would see that we call memcpy with ebp-8 address and the number of bytes to copy is 8. In pseudo-code it would look like:
memcpy(ebp-8, 004020f4, 8);
33 00401800 push 8
33 00401802 push offset WrongIP!GS_ExceptionPointers+0x8 (004020f4)
33 00401807 lea edx,[ebp-8]
33 0040180a push edx
33 0040180b call WrongIP!memcpy (00401010)
33 00401810 add esp,0Ch
If we examine 004020f4 address we would see that it contains 8 zeroes:
0:001> db 004020f4 l8
004020f4 00 00 00 00 00 00 00 00
Therefore memcpy overwrites our local variables that contain a jump address with zeroes. This explains why we have jumped to 0 address and why EIP was zero.
Finally our reconstructed stack trace looks like this:
WrongIP!internal_func_2+offset ; here we jump
WrongIP!internal_func_1+0x1f
WrongIP!thread_request+0xc
msvcr80!_endthread+0x4b
msvcr80!_endthread+0xcb
kernel32!BaseThreadStart+0x34
This was based on the fact that ESP was valid. If we have a zero or invalid ESP we can look at the entire raw stack range from thread environment block (TEB). Use !teb command to get thread stack range. In my example this command doesn’t work due to the lack of proper MS symbols but it reports TEB address and we can dump it:
0:001> !teb
TEB at 7efda000
error InitTypeRead( TEB )...
0:001> dd 7efda000 l3
7efda000 0069ffa4 006a0000 0069e000
Usually the second double word is the stack limit and the third is the stack base address so we can dump the range and start reconstructing stack trace for our example from the bottom of the stack (BaseThreadStart) or look after exception handling calls (shown in red):
0:001> dds 0069e000 006a0000
0069e000 00000000
0069e004 00000000
...
...
...
0069fb24 7d535b43 kernel32!UnhandledExceptionFilter+0×851
…
…
…
0069fbb0 0069fc20
0069fbb4 7d6354c9 ntdll!RtlDispatchException+0×11f
0069fbb8 0069fc38
0069fbbc 0069fc88
0069fc1c 00000000
0069fc20 00000000
0069fc24 7d61dd26 ntdll!NtRaiseException+0×12
0069fc28 7d61ea51 ntdll!KiUserExceptionDispatcher+0×29
0069fc2c 0069fc38
…
…
…
0069ff38 00000000
0069ff3c 00000000
0069ff40 00000000
0069ff44 00000000
0069ff48 00000000
0069ff4c 00000000
0069ff50 00000000
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000
0069ff60 0069ff70
0069ff64 0040188f WrongIP!internal_func_1+0×1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401850 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 004018ac WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
0069ff80 78132848 msvcr80!_endthread+0×4b
0069ff84 00401830 WrongIP!jump_func
0069ff88 6d5ba283
0069ff8c 00000000
0069ff90 00000000
0069ff94 00595620
0069ff98 c0000005
0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac 152916af
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0×34
0069ffc0 00595620
0069ffc4 00000000
…
…
…
- Dmitry Vostokov @ DumpAnalysis.org -
