Crash Dump Analysis

My friend was typing a long message in IE to one of his old schoolmates that he had just found on Internet. He spent about an hour writing and rewriting and when finally hit the Send button he got a page saying that connection was probably lost. Going back in URL history brought the empty edit box and all data was lost. Or was it? He called me and I immediately advised him to save a crash dump of iexplore.exe using Task Manager (Vista). I also asked him for a word he used to start his message. It was “Hello” in Russian. I got his dump file and opened it in WinDbg. Because the language of his message was Russian I assumed that it was still there in local buffers or heap entries in UNICODE format so I typed “ello” in Notepad and saved this in a Unicode text file. Loading it in a binary editor (I used Visual C++) showed the following sequence of bytes:

40 04 38 04 32 04 35 04 42 04

Then I did a search in WinDbg for this sequence from the first loaded module address till the end of user space:

0:000> lm
start    end        module name
003c0000 0045b000   iexplore
[...]

0:000> s 003c0000 L?7FFFFFFF 40 04 38 04 32 04 35 04 42 04
[...]
048971e4 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04  @.8.2.5.B.,. ...
[...]
08530fe4 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04 @.8.2.5.B.,. ...
[...]
201ea65c 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04 @.8.2.5.B.,. ...
[...] 

The number of found entries was big and I decided to output every entry into a file using the following script: 

.foreach ( address { s-[1]b 003c0000 L?7FFFFFFF 40 04 38 04 32 04 35 04 42 04 }) {.writemem c:\dmitry\ieout${address}.txt ${address}-10 ${address}+1000}

I got numerous files:

C:\dmitry>dir ieout*.txt
[...]
09/06/2008  08:53               4112 ieout0x048971e4.txt
09/06/2008  08:53               4112 ieout0x0489784c.txt
09/06/2008  08:53               4112 ieout0x0489b854.txt
09/06/2008  08:53               4112 ieout0x0489bc5c.txt
[...]

I combined all of them into one big file and sent it to my friend:

C:\dmitry>type ieout0x*.txt >ieoutall.txt

The file contained not only the final message but all intermediate typing histories too. He was very happy.

- Dmitry Vostokov @ DumpAnalysis.org -

This is the next scheduled book from Crash Dump Analysis Publishing Roadmap:

• Title: Windows® Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
• Author: Dmitry Vostokov
• Publisher: Opentask (1 September 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

• ISBN-13: 978-0-9558328-5-7
• Hardcover (Cloth): 256 pages

• ISBN-13: 978-1-906717-00-1
• Paperback: 256 pages

Draft Table of Contents will be published next month together with a sample chapter.

- Dmitry Vostokov @ DumpAnalysis.org -

Due to demand from people that prefer ebooks I published Memory Dump Analysis Anthology, Volume 1 in a digital format that can be purchased in Crash Dump Analysis Store. This format has color pictures inside.

- Dmitry Vostokov @ DumpAnalysis.org -

I’m very proud to announce that it is finally available in both paperback and hardback. Why have I made available both editions? Because I personally prefer hardcover books. You can order the book today and it will be printed in 3-5 days (paperback) or 5-10 days (hardcover) and sent to you:

Memory Dump Analysis Anthology, Volume 1

Note: although listed on Amazon and other online bookstores it is not immediately available at these stores at the moment due to the late submission. I apologize for this. However, I expect that in a few weeks pre-orders taken there will be eventually fulfilled. In the mean time, if you want the book now, you can use the link above.

- Dmitry Vostokov @ DumpAnalysis.org -

Although the first volume has not been published yet (scheduled for 15th of April, 2008) the planning for the second volume has already begun. Preliminary information is:

• Title: Memory Dump Analysis Anthology, Volume 2
• Paperback: 512 pages (*)
• ISBN-13: 978-0-9558328-7-1
• Author: Dmitry Vostokov
• Publisher: Opentask (01 Nov 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

Hardcover version is also planned. PDF version will be available for download too.

(*) subject to change

- Dmitry Vostokov @ DumpAnalysis.org -

This is a forthcoming reference book for technical support and escalation engineers troubleshooting and debugging complex software issues. The book is also invaluable for software maintenance and development engineers debugging unmanaged, managed and native code.

• Title: Windows® Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
• Author: Dmitry Vostokov
• Hardcover: 256 pages
• ISBN-13: 978-0-9558328-5-7
• Publisher: Opentask (1 July 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

- Dmitry Vostokov @ DumpAnalysis.org -

It is very easy to become a publisher nowadays. Much easier than I thought. I registered myself as a publisher under the name of OpenTask which is my registered business name in Ireland. I also got the list of ISBN numbers and therefore can announce product details for the first volume of Memory Dump Analysis Anthology series:

Memory Dump Analysis Anthology, Volume 1

• Paperback: 720 pages (*)
• ISBN-13: 978-0-9558328-0-2

• Hardcover: 720 pages (*)
• ISBN-13: 978-0-9558328-1-9

• Author: Dmitry Vostokov
• Publisher: Opentask (15 Apr 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

(*) subject to change 

PDF file will be available for download too.

- Dmitry Vostokov @ DumpAnalysis.org -

I have published the sample chapter from “Windows® Crash Dump Analysis” book. I have decided that it should be a short introduction to the scripting language from Debugging Tools for Windows package. If you know some C-style language like C, C++, Java or C# then the mapping between it and scripting facilities is pretty straightforward. You can download the chapter from this link:

Sample Chapter

- Dmitry Vostokov @ DumpAnalysis.org -

In the previous part I used WinDbg scripting to get raw stack data from user process dump. However the script needs to be modified if the dump is complete memory dump. Here I use !for_each_thread WinDbg extension command to dump stack trace and user space raw stack data for all threads except system threads because they don’t have user space stack counterpart and their TEB address is NULL:

!for_each_thread ".thread /r /p @#Thread; .if (@$teb != 0) {!thread @#Thread; r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2}"

We need to open a log file. It will be huge and we might want to dump raw stack contents for specific process only. In such case we can filter the output of the script using $proc pseudo-register, the address of EPROCESS:

!for_each_thread ".thread /r /p @#Thread; .if (@$teb != 0 & @$proc == <EPROCESS>) {!thread @#Thread; r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2}"

For example:

1: kd>!process 0 0
...
...
...
PROCESS 8596f9c8  SessionId: 0  Cid: 0fac    Peb: 7ffde000  ParentCid: 0f3c
    DirBase: 3fba6520  ObjectTable: d6654e28  HandleCount: 389.
    Image: explorer.exe
…
…
…

1: kd> !for_each_thread ".thread /r /p @#Thread; .if (@$teb != 0 & @$proc == 8596f9c8) {!thread @#Thread; r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2}”
Implicit thread is now 8659b208
Implicit process is now 8659b478
Loading User Symbols

Implicit thread is now 86599db0
Implicit process is now 8659b478
Loading User Symbols

...
...
...
Implicit thread is now 85b32db0
Implicit process is now 8596f9c8
Loading User Symbols

THREAD 85b32db0  Cid 0fac.0fb0  Teb: 7ffdd000 Win32Thread: bc0a6be8 WAIT: (Unknown) UserMode Non-Alertable
    859bda20  SynchronizationEvent
Not impersonating
DeviceMap                 d743e440
Owning Process            8596f9c8       Image:         explorer.exe
Wait Start TickCount      376275         Ticks: 102 (0:00:00:01.593)
Context Switch Count      3509                 LargeStack
UserTime                  00:00:00.078
KernelTime                00:00:00.203
Win32 Start Address Explorer!ModuleEntry (0x010148a4)
Start Address kernel32!BaseProcessStartThunk (0x77e617f8)
Stack Init ba5fe000 Current ba5fdc50 Base ba5fe000 Limit ba5f9000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
ba5fdc68 80833465 85b32db0 85b32e58 00000000 nt!KiSwapContext+0x26
ba5fdc94 80829a62 00000000 bc0a6be8 00000000 nt!KiSwapThread+0x2e5
ba5fdcdc bf89abe3 859bda20 0000000d 00000001 nt!KeWaitForSingleObject+0x346
ba5fdd38 bf89da53 000024ff 00000000 00000001 win32k!xxxSleepThread+0x1be
ba5fdd4c bf89e411 000024ff 00000000 0007fef8 win32k!xxxRealWaitMessageEx+0x12
ba5fdd5c 8088978c 0007ff08 7c8285ec badb0d00 win32k!NtUserWaitMessage+0x14
ba5fdd5c 7c8285ec 0007ff08 7c8285ec badb0d00 nt!KiFastCallEntry+0xfc (TrapFrame @ ba5fdd64)
0007feec 7739bf53 7c92addc 77e619d1 000d9298 ntdll!KiFastSystemCallRet
0007ff08 7c8fadbd 00000000 0007ff5c 0100fff1 USER32!NtUserWaitMessage+0xc
0007ff14 0100fff1 000d9298 7ffde000 0007ffc0 SHELL32!SHDesktopMessageLoop+0x24
0007ff5c 0101490c 00000000 00000000 000207fa Explorer!ExplorerWinMain+0x2c4
0007ffc0 77e6f23b 00000000 00000000 7ffde000 Explorer!ModuleEntry+0x6d
0007fff0 00000000 010148a4 00000000 78746341 kernel32!BaseProcessStart+0x23

Last set context:
TEB at 7ffdd000
    ExceptionList:        0007ffe0
    StackBase:            00080000
    StackLimit:           00072000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdd000
    EnvironmentPointer:   00000000
    ClientId:             00000fac . 00000fb0
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffde000
    LastErrorValue:       6
    LastStatusValue:      c0000008
    Count Owned Locks:    0
    HardErrorMode:        0
00072000  ????????
00072004  ????????
00072008  ????????
0007200c  ????????
00072010  ????????
00072014  ????????
00072018  ????????
0007201c  ????????
...
...
...
00079ff8  ????????
00079ffc  ????????
0007a000  00000000
0007a004  00000000
0007a008  00000000
0007a00c  00000000
0007a010  00000000
0007a014  00000000
0007a018  00000000
0007a01c  00000000
0007a020  00000000
0007a024  00000000
0007a028  00000000
0007a02c  00000000
...
...
...
0007ff04  0007ff14
0007ff08  0007ff14
0007ff0c  7c8fadbd SHELL32!SHDesktopMessageLoop+0x24
0007ff10  00000000
0007ff14  0007ff5c
0007ff18  0100fff1 Explorer!ExplorerWinMain+0x2c4
0007ff1c  000d9298
0007ff20  7ffde000
0007ff24  0007ffc0
0007ff28  00000000
0007ff2c  0007fd28
0007ff30  0007ff50
0007ff34  7ffde000
0007ff38  7c82758b ntdll!ZwQueryInformationProcess+0xc
0007ff3c  77e6c336 kernel32!GetErrorMode+0x18
0007ff40  ffffffff
0007ff44  0000000c
0007ff48  00000000
0007ff4c  00018fb8
0007ff50  000000ec
0007ff54  00000001
0007ff58  000d9298
0007ff5c  0007ffc0
0007ff60  0101490c Explorer!ModuleEntry+0x6d
0007ff64  00000000
0007ff68  00000000
0007ff6c  000207fa
0007ff70  00000001
0007ff74  00000000
0007ff78  00000000
0007ff7c  00000044
0007ff80  0002084c
0007ff84  0002082c
0007ff88  000207fc
0007ff8c  00000000
0007ff90  00000000
0007ff94  00000000
0007ff98  00000000
0007ff9c  f60e87fc
0007ffa0  00000002
0007ffa4  021a006a
0007ffa8  00000001
0007ffac  00000001
0007ffb0  00000000
0007ffb4  00000000
0007ffb8  00000000
0007ffbc  00000000
0007ffc0  0007fff0
0007ffc4  77e6f23b kernel32!BaseProcessStart+0x23
0007ffc8  00000000
0007ffcc  00000000
0007ffd0  7ffde000
0007ffd4  00000000
0007ffd8  0007ffc8
0007ffdc  b9a94ce4
0007ffe0  ffffffff
0007ffe4  77e61a60 kernel32!_except_handler3
0007ffe8  77e6f248 kernel32!`string'+0x88
0007ffec  00000000
0007fff0  00000000
0007fff4  00000000
0007fff8  010148a4 Explorer!ModuleEntry
0007fffc  00000000
00080000  78746341
...
...
...

Because complete memory dumps contain only physical memory contents some pages of raw stack data can be in page files and therefore unavailable.

- Dmitry Vostokov @ DumpAnalysis.org -

Small Memory Dumps, also referred as minidumps because they are stored in %SystemRoot% \ Minidump folder, contain only bugcheck information, kernel mode stack data and the list of loaded drivers. They can be used to transmit system crash information to a vendor or a 3rd-party for an automated crash dump analysis. Another use is to keep system crash history. In this part I discuss the scripting approach to extract information from all minidumps stored on a particular system. The script processes all minidump files and creates text log files containing the following information:

1. Crash dump name and type 

2. OS information, crash time and system uptime 

3. Processor context (r) and verbose stack trace (kv) prior to applying !analyze -v. This is useful sometimes when WinDbg reconstructs a different stack trace after changing a processor context to the execution context at the time of a trap, exception or fault. 

4. The output of !analyze -v command

5. Processor context (r) and verbose stack trace (kv) after !analyze -v command.

6. Code disassembly for the current execution pointer (EIP or x64 RIP). This includes forward (u) and backward (ub) disassembly, and we also try to disassemble the whole function (uf) which should succeed if we have symbol information

7. Raw stack dump with symbol information (dps)

8. The same raw stack data but interpreted as pointers to Unicode zero-terminated strings (dpu). Some pointers on the stack might point to local string buffers located on the same stack. This can be a slow operation and WinDbg might temporarily hang.

9. The same raw stack data but interpreted as pointers to ASCII zero-terminated strings (dpa). This can be a slow operation and WinDbg might temporarily hang.

10. Verbose information about loaded drivers (lmv)

11. CPU, machine ID, machine-specific registers, and verbose SMBIOS information like motherboard and devices (!sysinfo)

Here is WinDbg script listing:

$$
$$ MiniDmp2Txt: Dump information from minidump into log
$$
.logopen /d /u
.echo "command> ||"
||
.echo "command> vertarget"
vertarget
.echo "command> r (before analysis)"
r
.echo "command> kv (before analysis)"
kv 100
.echo "command> !analyze -v"
!analyze -v
.echo "command> r"
r
.echo "command> kv"
kv 100
.echo "command> ub eip"
ub eip
.echo "command> u eip"
u eip
.echo "command> uf eip"
uf eip
.echo "command> dps esp-3000 esp+3000"
dps esp-3000 esp+3000
.echo "command> dpu esp-3000 esp+3000"
dpu esp-3000 esp+3000
.echo "command> dpa esp-3000 esp+3000"
dpa esp-3000 esp+3000
.echo "command> lmv"
lmv
.echo "command> !sysinfo cpuinfo"
!sysinfo cpuinfo
.echo "command> !sysinfo cpuspeed"
!sysinfo cpuspeed
.echo "command> !sysinfo cpumicrocode"
!sysinfo cpumicrocode
.echo "command> !sysinfo gbl"
!sysinfo gbl
.echo "command> !sysinfo machineid"
!sysinfo machineid
.echo "command> !sysinfo registers"
!sysinfo registers
.echo "command> !sysinfo smbios -v"
!sysinfo smbios -v
.logclose
$$
$$ MiniDmp2Txt: End of File
$$

To run WinDbg automatically against each minidump file (.dmp) use the following VB script (customize symbol search path (-y) to point to your own folders):

'
' MiniDumps2Txt.vbs
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set Folder = fso.GetFolder(".")
Set Files = Folder.Files
Set WshShell = CreateObject("WScript.Shell")
For Each File In Files
  If Mid(File.Name,Len(File.Name)-3,4) = ".dmp" Then
    Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for Windows\WinDbg.exe -y ""srv*c:\ms*http://msdl.microsoft.com/download/symbols"" -z " + File.Name + " -c ""$$><c:\scripts\MiniDmp2Txt.txt;q"" -Q -QS -QY -QSY")
    Do While oExec.Status = 0
      WScript.Sleep 1000
    Loop
  End If
Next
'
' MiniDumps2Txt.vbs: End of File
'

We can also use kd.exe instead of WinDbg but its window will be hidden if we use the same VB script.

Log file interpretation is the subject of the next minidump analysis part.

- Dmitry Vostokov @ DumpAnalysis.org -