Analyzing Dr. Watson logs
Sunday, April 8th, 2007The main problem with Dr. Watson logs is lack of symbol information but this can be alleviated by using WinDbg if you have the same binary that crashed and produced the log entry. I’m going to illustrate this by using TestDefaultDebugger tool. Its main purpose is to crash. I use this tool here just to show you how to reconstruct stack trace.
If you run it and Dr. Watson is your default postmortem debugger you will get this event recoded in your Dr. Watson log:
*** ERROR: Module load completed but symbols could not be loaded for C:\Work\TestDefaultDebugger.exe
function: TestDefaultDebugger
004014e6 cc int 3
004014e7 cc int 3
004014e8 cc int 3
004014e9 cc int 3
004014ea cc int 3
004014eb cc int 3
004014ec cc int 3
004014ed cc int 3
004014ee cc int 3
004014ef cc int 3
FAULT ->004014f0 c7050000000000000000 mov dword ptr ds:[0],0 ds:0023:00000000=????????
004014fa c3 ret
004014fb cc int 3
004014fc cc int 3
004014fd cc int 3
004014fe cc int 3
004014ff cc int 3
00401500 0fb7542404 movzx edx,word ptr [esp+4]
00401505 89542404 mov dword ptr [esp+4],edx
00401509 e98e1c0000 jmp TestDefaultDebugger+0×319c (0040319c)
0040150e cc int 3
*—-> Stack Back Trace <----*
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
TestDefaultDebugger+0x14f0
TestDefaultDebugger+0x3470
TestDefaultDebugger+0x2a27
TestDefaultDebugger+0x8e69
TestDefaultDebugger+0x98d9
TestDefaultDebugger+0x6258
TestDefaultDebugger+0x836d
You see that when the log entry was saved there were no symbols available and this is the most common case. If you have such a log and no corresponding user dump (perhaps it was overwritten) then you can still reconstruct stack trace. To do this run WinDbg, set path to your application symbol files and load your application as a crash dump:
Microsoft (R) Windows Debugger Version 6.6.0007.5
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Work\TestDefaultDebugger.exe]
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols;c:\work
Executable search path is:
ModLoad: 00400000 00435000 C:\Work\TestDefaultDebugger.exe
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000000
eip=0040e8bb esp=00000000 ebp=00000000 iopl=0 nv up di pl nz na po nc
cs=0000 ss=0000 ds=0000 es=0000 fs=0000 gs=0000 efl=00000000
TestDefaultDebugger!wWinMainCRTStartup:
0040e8bb e876440000 call TestDefaultDebugger!__security_init_cookie (00412d36)
Now use ln command to find the nearest symbol:
0:000> ln TestDefaultDebugger+0×14f0
c:\testdefaultdebugger\testdefaultdebuggerdlg.cpp(155)
(004014f0) TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1 | (00401500) TestDefaultDebugger!CDialog::Create
Exact matches:
TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1 (void)
0:000> ln TestDefaultDebugger+0×3470
f:\rtm\vctools\vc7libs\ship\atlmfc\src\mfc\cmdtarg.cpp(381)+0×18
(00403358) TestDefaultDebugger!CCmdTarget::OnCmdMsg+0×118 | (00403472) TestDefaultDebugger!CCmdTarget::IsInvokeAllowed
0:000> ln TestDefaultDebugger+0×2a27
f:\rtm\vctools\vc7libs\ship\atlmfc\src\mfc\dlgcore.cpp(85)+0×17
(00402a0c) TestDefaultDebugger!CDialog::OnCmdMsg+0×1b | (00402a91) TestDefaultDebugger!CDialog::`scalar deleting destructor’
0:000> ln TestDefaultDebugger+0×8e69
f:\rtm\vctools\vc7libs\ship\atlmfc\src\mfc\wincore.cpp(2299)+0xd
(00408dd9) TestDefaultDebugger!CWnd::OnCommand+0×90 | (00408e70) TestDefaultDebugger!CWnd::OnNotify
0:000> ln TestDefaultDebugger+0×98d9
f:\rtm\vctools\vc7libs\ship\atlmfc\src\mfc\wincore.cpp(1755)+0xe
(004098a3) TestDefaultDebugger!CWnd::OnWndMsg+0×36 | (00409ecf) TestDefaultDebugger!CWnd::ReflectChildNotify
0:000> ln TestDefaultDebugger+0×6258
f:\rtm\vctools\vc7libs\ship\atlmfc\src\mfc\wincore.cpp(1741)+0×17
(00406236) TestDefaultDebugger!CWnd::WindowProc+0×22 | (0040627a) TestDefaultDebugger!CTestCmdUI::CTestCmdUI
0:000> ln TestDefaultDebugger+0×836d
f:\rtm\vctools\vc7libs\ship\atlmfc\src\mfc\wincore.cpp(243)
(004082d3) TestDefaultDebugger!AfxCallWndProc+0×9a | (004083c0) TestDefaultDebugger!AfxWndProc
So we reconstructed the stack trace:
TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1
TestDefaultDebugger!CCmdTarget::OnCmdMsg+0×118
TestDefaultDebugger!CDialog::OnCmdMsg+0×1b
TestDefaultDebugger!CWnd::OnCommand+0×90
TestDefaultDebugger!CWnd::OnWndMsg+0×36
TestDefaultDebugger!CWnd::WindowProc+0×22
TestDefaultDebugger!AfxCallWndProc+0×9a
To check it we disassemble the top and see that it corresponds to our crash point from Dr. Watson log:
0:000> u TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1
TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1 [c:\testdefaultdebugger\testdefaultdebuggerdlg.cpp @ 155]:
004014f0 c7050000000000000000 mov dword ptr ds:[0],0
004014fa c3 ret
004014fb cc int 3
004014fc cc int 3
004014fd cc int 3
004014fe cc int 3
004014ff cc int 3
Although I haven’t tried it yet I believe you can also apply this technique to old Windows 98 or Windows Me Dr. Watson logs.
- Dmitry Vostokov -
