Jeffrey Richter updates his book “Programming Applications for Microsoft Windows” to include information about Windows XP and Vista. The new book title is ”Windows via C/C++”. Looking forward to reading this classic book again.
- Dmitry Vostokov @ DumpAnalysis.org -
Small Memory Dumps, also referred as minidumps because they are stored in %SystemRoot% \ Minidump folder, contain only bugcheck information, kernel mode stack data and the list of loaded drivers. They can be used to transmit system crash information to a vendor or a 3rd-party for an automated crash dump analysis. Another use is to keep system crash history. In this part I discuss the scripting approach to extract information from all minidumps stored on a particular system. The script processes all minidump files and creates text log files containing the following information:
Crash dump name and type
OS information, crash time and system uptime
Processor context (r) and verbose stack trace (kv) prior to applying !analyze -v. This is useful sometimes when WinDbg reconstructs a different stack trace after changing a processor context to the execution context at the time of a trap, exception or fault.
The output of !analyze -v command
Processor context (r) and verbose stack trace (kv) after !analyze -v command.
Code disassembly for the current execution pointer (EIP or x64 RIP). This includes forward (u) and backward (ub) disassembly, and we also try to disassemble the whole function (uf) which should succeed if we have symbol information
Raw stack dump with symbol information (dps)
The same raw stack data but interpreted as pointers to Unicode zero-terminated strings (dpu). Some pointers on the stack might point to local string buffers located on the same stack. This can be a slow operation and WinDbg might temporarily hang.
The same raw stack data but interpreted as pointers to ASCII zero-terminated strings (dpa). This can be a slow operation and WinDbg might temporarily hang.
Verbose information about loaded drivers (lmv)
CPU, machine ID, machine-specific registers, and verbose SMBIOS information like motherboard and devices (!sysinfo)
Here is WinDbg script listing:
$$
$$ MiniDmp2Txt: Dump information from minidump into log
$$
.logopen /d /u
.echo "command> ||"
||
.echo "command> vertarget"
vertarget
.echo "command> r (before analysis)"
r
.echo "command> kv (before analysis)"
kv 100
.echo "command> !analyze -v"
!analyze -v
.echo "command> r"
r
.echo "command> kv"
kv 100
.echo "command> ub eip"
ub eip
.echo "command> u eip"
u eip
.echo "command> uf eip"
uf eip
.echo "command> dps esp-3000 esp+3000"
dps esp-3000 esp+3000
.echo "command> dpu esp-3000 esp+3000"
dpu esp-3000 esp+3000
.echo "command> dpa esp-3000 esp+3000"
dpa esp-3000 esp+3000
.echo "command> lmv"
lmv
.echo "command> !sysinfo cpuinfo"
!sysinfo cpuinfo
.echo "command> !sysinfo cpuspeed"
!sysinfo cpuspeed
.echo "command> !sysinfo cpumicrocode"
!sysinfo cpumicrocode
.echo "command> !sysinfo gbl"
!sysinfo gbl
.echo "command> !sysinfo machineid"
!sysinfo machineid
.echo "command> !sysinfo registers"
!sysinfo registers
.echo "command> !sysinfo smbios -v"
!sysinfo smbios -v
.logclose
$$
$$ MiniDmp2Txt: End of File
$$
To run WinDbg automatically against each minidump file (.dmp) use the following VB script (customize symbol search path (-y) to point to your own folders):
'
' MiniDumps2Txt.vbs
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set Folder = fso.GetFolder(".")
Set Files = Folder.Files
Set WshShell = CreateObject("WScript.Shell")
For Each File In Files
If Mid(File.Name,Len(File.Name)-3,4) = ".dmp" Then
Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for Windows\WinDbg.exe -y ""srv*c:\ms*http://msdl.microsoft.com/download/symbols"" -z " + File.Name + " -c ""$$><c:\scripts\MiniDmp2Txt.txt;q"" -Q -QS -QY -QSY")
Do While oExec.Status = 0
WScript.Sleep 1000
Loop
End If
Next
'
' MiniDumps2Txt.vbs: End of File
'
We can also use kd.exe instead of WinDbg but its window will be hidden if we use the same VB script.
Log file interpretation is the subject of the next minidump analysis part.
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes we need to dump the whole thread stack data to find traces of hooks, printer drivers or just string fragments. This is usually done by finding the appropriate TEB and dumping the data between StackLimit and StackBase addresses, for example:
0:000> ~
. 0 Id: 106c.4e4 Suspend: 1 Teb: 7ffde000 Unfrozen
1 Id: 106c.4e0 Suspend: 1 Teb: 7ffdc000 Unfrozen
2 Id: 106c.1158 Suspend: 1 Teb: 7ffdb000 Unfrozen
3 Id: 106c.c3c Suspend: 1 Teb: 7ffd9000 Unfrozen
4 Id: 106c.1174 Suspend: 1 Teb: 7ffd8000 Unfrozen
5 Id: 106c.1168 Suspend: 1 Teb: 7ffd4000 Unfrozen
6 Id: 106c.1568 Suspend: 1 Teb: 7ffaf000 Unfrozen
7 Id: 106c.1574 Suspend: 1 Teb: 7ffad000 Unfrozen
8 Id: 106c.964 Suspend: 1 Teb: 7ffac000 Unfrozen
9 Id: 106c.1164 Suspend: 1 Teb: 7ffab000 Unfrozen
10 Id: 106c.d84 Suspend: 1 Teb: 7ffaa000 Unfrozen
11 Id: 106c.bf4 Suspend: 1 Teb: 7ffa9000 Unfrozen
12 Id: 106c.eac Suspend: 1 Teb: 7ffa8000 Unfrozen
13 Id: 106c.614 Suspend: 1 Teb: 7ffd5000 Unfrozen
14 Id: 106c.cd8 Suspend: 1 Teb: 7ffa7000 Unfrozen
15 Id: 106c.1248 Suspend: 1 Teb: 7ffa6000 Unfrozen
16 Id: 106c.12d4 Suspend: 1 Teb: 7ffa4000 Unfrozen
17 Id: 106c.390 Suspend: 1 Teb: 7ffa3000 Unfrozen
18 Id: 106c.764 Suspend: 1 Teb: 7ffa1000 Unfrozen
19 Id: 106c.f48 Suspend: 1 Teb: 7ff5f000 Unfrozen
20 Id: 106c.14a8 Suspend: 1 Teb: 7ff53000 Unfrozen
21 Id: 106c.464 Suspend: 1 Teb: 7ff4d000 Unfrozen
22 Id: 106c.1250 Suspend: 1 Teb: 7ffa5000 Unfrozen
23 Id: 106c.fac Suspend: 1 Teb: 7ff5c000 Unfrozen
24 Id: 106c.1740 Suspend: 1 Teb: 7ffd7000 Unfrozen
25 Id: 106c.ae4 Suspend: 1 Teb: 7ffd6000 Unfrozen
26 Id: 106c.a4c Suspend: 1 Teb: 7ffdd000 Unfrozen
27 Id: 106c.1710 Suspend: 1 Teb: 7ffda000 Unfrozen
28 Id: 106c.1430 Suspend: 1 Teb: 7ffa2000 Unfrozen
29 Id: 106c.1404 Suspend: 1 Teb: 7ff4e000 Unfrozen
30 Id: 106c.9a8 Suspend: 1 Teb: 7ff4c000 Unfrozen
31 Id: 106c.434 Suspend: 1 Teb: 7ff4b000 Unfrozen
32 Id: 106c.c8c Suspend: 1 Teb: 7ff4a000 Unfrozen
33 Id: 106c.4f0 Suspend: 1 Teb: 7ff49000 Unfrozen
34 Id: 106c.be8 Suspend: 1 Teb: 7ffae000 Unfrozen
35 Id: 106c.14e0 Suspend: 1 Teb: 7ff5d000 Unfrozen
36 Id: 106c.fe0 Suspend: 1 Teb: 7ff5b000 Unfrozen
37 Id: 106c.1470 Suspend: 1 Teb: 7ff57000 Unfrozen
38 Id: 106c.16c4 Suspend: 1 Teb: 7ff5e000 Unfrozen
0:000> !teb 7ffad000
TEB at 7ffad000
ExceptionList: 0181ff0c
StackBase: 01820000
StackLimit: 0181c000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffad000
EnvironmentPointer: 00000000
ClientId: 0000106c . 00001574
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: c000000d
Count Owned Locks: 0
HardErrorMode: 0
0:000> dps 0181c000 01820000
0181c000 00000000
0181c004 00000000
0181c008 00000000
0181c00c 00000000
0181c010 00000000
0181c014 00000000
0181c018 00000000
0181c01c 00000000
0181c020 00000000
0181c024 00000000
...
...
...
0181ffb8 0181ffec
0181ffbc 77e6608b kernel32!BaseThreadStart+0x34
0181ffc0 00f31eb0
0181ffc4 00000000
0181ffc8 00000000
0181ffcc 00f31eb0
0181ffd0 8a38f7a8
0181ffd4 0181ffc4
0181ffd8 88a474b8
0181ffdc ffffffff
0181ffe0 77e6b7d0 kernel32!_except_handler3
0181ffe4 77e66098 kernel32!`string'+0x98
0181ffe8 00000000
0181ffec 00000000
0181fff0 00000000
0181fff4 7923a709
0181fff8 00f31eb0
0181fffc 00000000
01820000 ????????
However, if our process has many threads, like in the example above, and we want to dump stack data from all of them, we need to automate this process. After several attempts I created the following simple script which can be copy-pasted into WinDbg command window or saved in a text file to be loaded and executed later via WinDbg $$>< command. The script takes the advantage of the following command
~e (Thread-Specific Command)
The ~e command executes one or more commands for a specific thread or for all threads in the target process.
(from WinDbg help)
Here is the script:
~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2
Raw stack data from different stacks is separated by !teb output for clarity, for example:
0:000> .logopen rawdata.log
0:000> ~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2
TEB at 7ffde000
ExceptionList: 0007fd38
StackBase: 00080000
StackLimit: 0007c000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffde000
EnvironmentPointer: 00000000
ClientId: 0000106c . 000004e4
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
0007c000 00000000
0007c004 00000000
0007c008 00000000
0007c00c 00000000
0007c010 00000000
0007c014 00000000
0007c018 00000000
0007c01c 00000000
0007c020 00000000
0007c024 00000000
...
...
...
...
...
...
...
0977ffb4 00000000
0977ffb8 0977ffec
0977ffbc 77e6608b kernel32!BaseThreadStart+0x34
0977ffc0 025c3728
0977ffc4 00000000
0977ffc8 00000000
0977ffcc 025c3728
0977ffd0 a50c4963
0977ffd4 0977ffc4
0977ffd8 000a5285
0977ffdc ffffffff
0977ffe0 77e6b7d0 kernel32!_except_handler3
0977ffe4 77e66098 kernel32!`string'+0x98
0977ffe8 00000000
0977ffec 00000000
0977fff0 00000000
0977fff4 77bcb4bc msvcrt!_endthreadex+0x2f
0977fff8 025c3728
0977fffc 00000000
09780000 ????????
TEB at 7ffae000
ExceptionList: 0071ff64
StackBase: 00720000
StackLimit: 0071c000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffae000
EnvironmentPointer: 00000000
ClientId: 0000106c . 00000be8
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: c000000d
Count Owned Locks: 0
HardErrorMode: 0
0071c000 00000000
0071c004 00000000
0071c008 00000000
0071c00c 00000000
0071c010 00000000
0071c014 00000000
0071c018 00000000
0071c01c 00000000
0071c020 00000000
0071c024 00000000
0071c028 00000000
0071c02c 00000000
0071c030 00000000
0071c034 00000000
0071c038 00000000
0071c03c 00000000
0071c040 00000000
0071c044 00000000
0071c048 00000000
0071c04c 00000000
0071c050 00000000
0071c054 00000000
...
...
...
...
...
...
...
0:000> .logclose
Instead of (or in addition to) dps command used in the script we can use dpu or dpa commands to dump all strings that are pointed to by stack data or create an even more complex script that does triple dereference.
- Dmitry Vostokov @ DumpAnalysis.org -
After blogging for more than a year, accumulating initial amount of material and being persuaded by my colleagues at Citrix I finally decided to embrace a new challenge and write a book with a simple working title: Windows® Crash Dump Analysis.
The reader will master crash and hang memory dump analysis for process, kernel and complete memory dumps. The book will provide answers to many questions including those in the following list:
Memory Dump Analysis Interview Questions
and will also include guidelines for implementing SMART (Smart Memory Analysis in Real Time) process in a technical support or software maintenance environment.
As this is my first book I’m going to write the draft version online. More details and the link to Table of Contents will be announced in September/October.
I’ll continue blogging about crash dump analysis at the same time.
- Dmitry Vostokov @ DumpAnalysis.org -
If you are developing and debugging user space applications (and/or doing crash dump analysis in user space) and you want to understand Windows kernel dumps and device drivers better (and probably start writing your own kernel tools) here is the reading list I found the most effective over the last 4 years:
0. Read and re-read Windows Internals book in parallel while reading all other books. I read all editions by the way. It will show you the big picture and some useful WinDbg commands and techniques but you need to read device driver books to fill the gaps and be confident in kernel space:
1. Start with “The Windows 2000 Device Driver Book: A Guide for Programmers (2nd Edition)”. This short book will show you the basics and you can start writing your drivers and kernel tools immediately.
2. Next read “Windows NT Device Driver Development” book to consolidate your knowledge. This book has been reprinted by OSR:
3. Don’t stop here. Read “Developing Windows NT Device Drivers:
A Programmer’s Handbook”. This is very good book explaining everything in great detail and good pictures. You will finally understand various buffering methods.
4. Continue with WDM drivers and modern presentation: “Programming the Microsoft Windows Driver Model, Second Edition”. Must read even if your drivers are not WDM.
5. Finally read “Developing Drivers with the Windows Driver Foundation” book as this is the future and it also covers ETW (event tracing for Windows), WinDbg extensions, PREfast and static driver verifier.
Additional reading (not including DDK Help which you will use anyway) can be done in parallel after finishing “Windows NT Device Driver Development” book:
1. OSR NT Insider articles. I have their full printed collection 1996 - 2006
2. “Windows NT File System Internals” reprinted by OSR:
3. “Rootkits: Subverting the Windows Kernel” book will show you Windows kernel from hacker perspective. In addition you will find overview of kernel areas not covered in other books.
Of course, you must know C language and its idioms really well. Really know it down to assembly language level! I’ll publish another reading list soon. Stay tuned.
- Dmitry Vostokov @ DumpAnalysis.org -
The following interview questions might be useful to assess the skill level in crash dump analysis on Windows platforms. These could be useful for debugging interviews as well.
What is FPO?
How many exceptions can be found in a crash dump?
You see the message from WinDbg:
WARNING: Stack unwind information not available. Following frames may be wrong.
What would you do?
How would you find spinlock implementation if you have a kernel dump?
What is OMAP?
What is full page heap?
Company name is missing from module information. How would you try to find it?
What is IDT?
How does a postmortem debugger work?
You’ve got a mini dump of your application. How would you disassemble the code?
Memory consumption is growing for an application. How would you discover the leaking component?
What is IRQL?
When do you use TEB?
You’ve got 200 process dumps from a server. You need to find a deadlock. How would you do it?
You’ve got a complete memory dump from a server. You need to find a deadlock. How would you do it?
What is GC heap?
Your customer is reluctant to send a dump due to security policies. What is your next step?
What is a first chance exception?
I’ve created a permanent page and will add more questions there in the future:
Memory Dump Analysis Interview Questions
- Dmitry Vostokov @ DumpAnalysis.org -
The Jobs section was created on Crash Dump Analysis Portal to assist companies in finding engineers skilled in crash/core dump analysis. Please read guidelines at:
http://www.dumpanalysis.org/index.php?q=jobs
- Dmitry Vostokov @ DumpAnalysis.org -
In contrast to Double Free pattern in a user mode process heap double free in a kernel mode pool results in immediate bugcheck in order to identify the driver causing the problem (BAD_POOL_CALLER bugcheck with Arg1 == 7):
2: kd> !analyze -v
...
...
...
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000007, Attempt to free pool which was already freed
Arg2: 0000121a, (reserved)
Arg3: 02140001, Memory contents of the pool block
Arg4: 89ba74f0, Address of the block of pool being deallocated
If we look at the block being deallocated we would see that it was marked as “Free” block:
2: kd> !pool 89ba74f0
Pool page 89ba74f0 region is Nonpaged pool
89ba7000 size: 270 previous size: 0 (Allocated) Thre (Protected)
89ba7270 size: 8 previous size: 270 (Free) ....
89ba7278 size: 18 previous size: 8 (Allocated) ReEv
89ba7290 size: 80 previous size: 18 (Allocated) Mdl
89ba7310 size: 80 previous size: 80 (Allocated) Mdl
89ba7390 size: 30 previous size: 80 (Allocated) Vad
89ba73c0 size: 98 previous size: 30 (Allocated) File (Protected)
89ba7458 size: 8 previous size: 98 (Free) Wait
89ba7460 size: 28 previous size: 8 (Allocated) FSfm
89ba74a0 size: 40 previous size: 18 (Allocated) Ntfr
89ba74e0 size: 8 previous size: 40 (Free) File
*89ba74e8 size: a0 previous size: 8 (Free ) *ABCD
Owning component : Unknown (update pooltag.txt)
89ba7588 size: 38 previous size: a0 (Allocated) Sema (Protected)
89ba75c0 size: 38 previous size: 38 (Allocated) Sema (Protected)
89ba75f8 size: 10 previous size: 38 (Free) Nbtl
89ba7608 size: 98 previous size: 10 (Allocated) File (Protected)
89ba76a0 size: 28 previous size: 98 (Allocated) Ntfn
89ba76c8 size: 40 previous size: 28 (Allocated) Ntfr
89ba7708 size: 28 previous size: 40 (Allocated) NtFs
89ba7730 size: 40 previous size: 28 (Allocated) Ntfr
89ba7770 size: 40 previous size: 40 (Allocated) Ntfr
89ba7a10 size: 270 previous size: 260 (Allocated) Thre (Protected)
89ba7c80 size: 20 previous size: 270 (Allocated) VadS
89ba7ca0 size: 40 previous size: 20 (Allocated) Ntfr
89ba7ce0 size: b0 previous size: 40 (Allocated) Ussy
89ba7d90 size: 270 previous size: b0 (Free) Thre
The pool tag is a 4 byte character sequence used to associate drivers with pool blocks and is useful to identify a driver allocated or freed a block. In our case the pool tag is ABCD and it is associated with the driver which previously freed the block. All known pool tags corresponding to kernel components can be found in pooltag.txt located in triage subfolder where you installed WinDbg. However our ABCD tag is not listed there. We can try to find the driver corresponding to ABCD tag using findstr CMD command:
C:\Windows\System32\drivers>findstr /m /l hABCD *.sys
The results of the search will help us to identify the driver which freed the block first. The driver which double freed the same block can be found from the call stack and it might be the same driver or a different driver:
2: kd> k
ChildEBP RetAddr
f78be910 8089c8f4 nt!KeBugCheckEx+0x1b
f78be978 8089c622 nt!ExFreePoolWithTag+0x477
f78be988 f503968b nt!ExFreePool+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
f78be990 f5024a6e driver+0×1768b
f78be9a0 f50249e7 driver+0×2a6e
f78be9a4 84b430e0 driver+0×29e7
Because we don’t have symbol files for driver.sys WinDbg warns us that it was unable to identify the correct stack trace and driver.sys might not have called ExFreePool or ExFreePoolWithTag. To verify that driver.sys called ExFreePool indeed we disassemble backwards the return address of ExFreePool:
2: kd> ub f503968b
driver+0×1767b:
f503967b 90 nop
f503967c 90 nop
f503967d 90 nop
f503967e 90 nop
f503967f 90 nop
f5039680 8b442404 mov eax,dword ptr [esp+4]
f5039684 50 push eax
f5039685 ff15202302f5 call dword ptr [driver+0×320 (f5022320)]
Finally we can get some info from the driver:
2: kd> lmv m driver
start end module name
f5022000 f503e400 driver (no symbols)
Loaded symbol image file: driver.sys
Image path: \SystemRoot\System32\drivers\driver.sys
Image name: driver.sys
Timestamp: Tue Aug 12 11:32:16 2007
If the company name developed the driver is absent we can try techniques outlined in Unknown Component pattern.
If we have symbols it is very easy to identify the code as can be seen from this 64-bit dump:
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 0000000000000007, Attempt to free pool which was already freed
Arg2: 000000000000121a, (reserved)
Arg3: 0000000000000080, Memory contents of the pool block
Arg4: fffffade6d54e270, Address of the block of pool being deallocated
0: kd> kL
fffffade`45517b08 fffff800`011ad905 nt!KeBugCheckEx
fffffade`45517b10 fffffade`5f5991ac nt!ExFreePoolWithTag+0x401
fffffade`45517bd0 fffffade`5f59a0b0 driver64!ProcessDataItem+0×198
fffffade`45517c70 fffffade`5f5885a6 driver64!OnDataArrival+0×2b4
fffffade`45517cd0 fffff800`01299cae driver64!ReaderThread+0×15a
fffffade`45517d70 fffff800`0102bbe6 nt!PspSystemThreadStartup+0×3e
fffffade`45517dd0 00000000`00000000 nt!KiStartSystemThread+0×16
- Dmitry Vostokov @ DumpAnalysis.org -
The HTML version of the presentation is located here:
Reading Windows-based Code (Part 5)
- Dmitry Vostokov @ DumpAnalysis.org -
Because Citrix acquires XenSource I suddenly became interested in live target debugging on that virtualization platform. Found the following public links:
How to convert VMWare image to Xen
How to use WinDbg to debug Windows targets under Xen
- Dmitry Vostokov @ DumpAnalysis.org -
Double-free bugs lead to Dynamic Memory Corruption. The reason why Double Free deserves its own pattern name is the fact that either debug runtime libraries or even OS itself detect such bugs and save crash dumps immediately.
For some heap implementations double free doesn’t lead to an immediate heap corruption and subsequent crash. For example, if you allocate 3 blocks in a row and then free the middle one twice there will be no crash as the second free call is able to detect that the block was already freed and does nothing. The following program loops forever and never crashes:
#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
while (true)
{
puts("Allocate: p1");
void *p1 = malloc(100);
puts("Allocate: p2");
void *p2 = malloc(100);
puts("Allocate: p3");
void *p3 = malloc(100);
puts("Free: p2");
free(p2);
puts(”Double-Free: p2″);
free(p2);
puts(”Free: p1″);
free(p1);
puts(”Free: p3″);
free(p3);
Sleep(100);
}
return 0;
}
The output of the program:
...
...
...
Allocate: p1
Allocate: p2
Allocate: p3
Free: p2
Double-Free: p2
Free: p1
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p2
Double-Free: p2
Free: p1
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p2
Double-Free: p2
...
...
...
However if a free call triggered heap coalescence (adjacent free blocks form the bigger free block) then we have a heap corruption crash on the next double-free call because the coalescence triggered by the previous free call erased free block information:
#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
while (true)
{
puts("Allocate: p1");
void *p1 = malloc(100);
puts("Allocate: p2");
void *p2 = malloc(100);
puts("Allocate: p3");
void *p3 = malloc(100);
puts("Free: p3");
free(p3);
puts("Free: p1");
free(p1);
puts("Free: p2");
free(p2);
puts(”Double-Free: p2″);
free(p2);
Sleep(100);
}
return 0;
}
The output of the program:
Allocate: p1
Allocate: p2
Allocate: p3
Free: p3
Free: p1
Free: p2
Double-Free: p2
Crash!
If we open a crash dump we would see the following stack trace:
0:000> r
eax=00922130 ebx=00920000 ecx=10101010 edx=10101010 esi=00922128 edi=00921fc8
eip=76ee1ad5 esp=0012fd6c ebp=0012fd94 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
ntdll!RtlpCoalesceFreeBlocks+0x6ef:
76ee1ad5 8b4904 mov ecx,dword ptr [ecx+4] ds:0023:10101014=????????
0:000> kL
ChildEBP RetAddr
0012fd94 76ee1d37 ntdll!RtlpCoalesceFreeBlocks+0x6ef
0012fe8c 76ee1c21 ntdll!RtlpFreeHeap+0x1e2
0012fea8 758d7a7e ntdll!RtlFreeHeap+0x14e
0012febc 6cff4c39 kernel32!HeapFree+0x14
0012ff08 0040107b msvcr80!free+0xcd
0012ff5c 004011f1 DoubleFree!wmain+0x7b
0012ffa0 758d3833 DoubleFree!__tmainCRTStartup+0x10f
0012ffac 76eba9bd kernel32!BaseThreadInitThunk+0xe
0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23
This is illustrated on the following picture where free calls result in heap coalescence and the subsequent double-free call corrupts the heap:
The problem here is that heap coalescence can be triggered some time after the double free so we need some solution to diagnose double-free bugs earlier, ideally at the first double-free call. For example, the following program crashes during the normal free operation long after the first double-free happened:
#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
while (true)
{
puts("Allocate: p1");
void *p1 = malloc(100);
puts("Allocate: p2");
void *p2 = malloc(100);
puts("Allocate: p3");
void *p3 = malloc(100);
puts("Free: p1");
free(p1);
puts("Free: p2");
free(p2);
puts(”Double-Free: p2″);
free(p2);
puts(”Double-Free: p3″);
free(p3);
Sleep(100);
}
return 0;
}
The output of the program:
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Free: p3
Crash!
If we enable full page heap using gflags.exe from Debugging Tools for Windows the program crashes immediately on the double free call:
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Crash!
The crash dump shows the following stack trace:
0:000> kL
ChildEBP RetAddr
0012f810 71aa4ced ntdll!DbgBreakPoint+0x1
0012f834 71aa9fc2 verifier!VerifierStopMessage+0x1fd
0012f890 71aaa4da verifier!AVrfpDphReportCorruptedBlock+0x102
0012f8a4 71ab2c98 verifier!AVrfpDphCheckNormalHeapBlock+0x18a
0012f8b8 71ab2a0e verifier!_EH4_CallFilterFunc+0x12
0012f8e0 76ee1039 verifier!_except_handler4+0x8e
0012f904 76ee100b ntdll!ExecuteHandler2+0x26
0012f9ac 76ee0e97 ntdll!ExecuteHandler+0x24
0012f9ac 71aaa3ad ntdll!KiUserExceptionDispatcher+0xf
0012fcf0 71aaa920 verifier!AVrfpDphCheckNormalHeapBlock+0x5d
0012fd0c 71aa879b verifier!AVrfpDphNormalHeapFree+0x20
0012fd60 76f31c8f verifier!AVrfDebugPageHeapFree+0x1cb
0012fda8 76efd9fa ntdll!RtlDebugFreeHeap+0x2f
0012fe9c 76ee1c21 ntdll!RtlpFreeHeap+0x5f
0012feb8 758d7a7e ntdll!RtlFreeHeap+0x14e
0012fecc 6cff4c39 kernel32!HeapFree+0x14
0012ff18 0040105f msvcr80!free+0xcd
0012ff5c 004011f1 DoubleFree!wmain+0x5f
0012ffa0 758d3833 DoubleFree!__tmainCRTStartup+0x10f
0012ffac 76eba9bd kernel32!BaseThreadInitThunk+0xe
0:000> !gflag
Current NtGlobalFlag contents: 0x02000000
hpa - Place heap allocations at ends of pages
If we enable heap free checking instead of page heap we get our crash on the first double free call immediately too:
Allocate: p1
Allocate: p2
Allocate: p3
Free: p1
Free: p2
Double-Free: p2
Crash!
The crash dump shows the following stack trace:
0:000> r
eax=feeefeee ebx=001b2040 ecx=001b0000 edx=001b2040 esi=d4476047 edi=001b2038
eip=76ee2086 esp=0012fe68 ebp=0012fe9c iopl=0 nv up ei ng nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010286
ntdll!RtlpLowFragHeapFree+0x31:
76ee2086 8b4604 mov eax,dword ptr [esi+4] ds:0023:d447604b=????????
0:000> kL
ChildEBP RetAddr
0012fe9c 76ee18c3 ntdll!RtlpLowFragHeapFree+0x31
0012feb0 758d7a7e ntdll!RtlFreeHeap+0x101
0012fec4 6cff4c39 kernel32!HeapFree+0x14
0012ff10 0040106d msvcr80!free+0xcd
0012ff5c 004011f1 DoubleFree!wmain+0x6d
0012ffa0 758d3833 DoubleFree!__tmainCRTStartup+0x10f
0012ffac 76eba9bd kernel32!BaseThreadInitThunk+0xe
0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23
0:000> !gflag
Current NtGlobalFlag contents: 0x00000020
hfc - Enable heap free checking
- Dmitry Vostokov @ DumpAnalysis.org -
Almost finished reading the book and wrote a short review:
Secure Programming with Static Analysis
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes the process size constantly grows but there is no difference in the process heap size. In such cases we need to check whether the process uses Microsoft .NET runtime (CLR). If one of the loaded modules is mscorwks.dll or mscorsvr.dll then it is most likely. Then we should check CLR heap statistics.
In .NET world dynamically allocated objects are garbage collected (GC) and therefore simple allocate-and-forget memory leaks are not possible. To simulate that I created the following C# program:
using System;
namespace CLRHeapLeak
{
class Leak
{
private byte[] m_data;
public Leak()
{
m_data = new byte[1024];
}
}
class Program
{
static void Main(string[] args)
{
Leak leak = new Leak();
while (true)
{
leak = new Leak();
System.Threading.Thread.Sleep(100);
}
}
}
}
If we run it the process size will never grow. GC thread will collect and free unreferenced Leak classes. This can be seen from inspecting memory dumps taken with userdump.exe after the start, 2, 6 and 12 minutes. The GC heap never grows higher than 1Mb and the number of CLRHeapLeak.Leak and System.Byte[] objects always fluctuates between 100 and 500. For example, on 12th minute we have the following statistics:
0:000> .loadby sos mscorwks
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0147160c
generation 1 starts at 0x0147100c
generation 2 starts at 0x01471000
ephemeral segment allocation context: (0x014dc53c, 0x014dd618)
segment begin allocated size
004aedb8 790d7ae4 790f7064 0x0001f580(128384)
01470000 01471000 014dd618 0x0006c618(443928)
Large object heap starts at 0x02471000
segment begin allocated size
02470000 02471000 02473250 0x00002250(8784)
Total Size 0x8dde8(581096)
------------------------------
GC Heap Size 0×8dde8(581096)
0:000> !dumpheap -stat
total 2901 objects
Statistics:
Count TotalSize Class Name
1 12 System.Security.Permissions.SecurityPermission
1 24 System.OperatingSystem
1 24 System.Version
1 24 System.Reflection.Assembly
1 28 System.SharedStatics
1 36 System.Int64[]
1 40 System.AppDomainSetup
3 60 System.RuntimeType
5 60 System.Object
2 72 System.Security.PermissionSet
1 72 System.ExecutionEngineException
1 72 System.StackOverflowException
1 72 System.OutOfMemoryException
1 100 System.AppDomain
7 100 Free
2 144 System.Threading.ThreadAbortException
4 328 System.Char[]
418 5016 CLRHeapLeak.Leak
5 8816 System.Object[]
2026 128632 System.String
418 433048 System.Byte[]
Total 2901 objects
However, if we make Leak objects always referenced by introducing the following changes into the program:
using System;
namespace CLRHeapLeak
{
class Leak
{
private byte[] m_data;
private Leak m_prevLeak;
public Leak()
{
m_data = new byte[1024];
}
public Leak(Leak prevLeak)
{
m_prevLeak = prevLeak;
m_data = new byte[1024];
}
}
class Program
{
static void Main(string[] args)
{
Leak leak = new Leak();
while (true)
{
leak = new Leak(leak);
System.Threading.Thread.Sleep(100);
}
}
}
}
then if we run the program we would see in Task Manager that it grows over time. Taking consecutive memory dumps after the start, 10 and 16 minutes, shows that Win32 heap segments have always the same size:
0:000> !heap 0 0
Index Address Name Debugging options enabled
1: 00530000
Segment at 00530000 to 00630000 (0003d000 bytes committed)
2: 00010000
Segment at 00010000 to 00020000 (00003000 bytes committed)
3: 00520000
Segment at 00520000 to 00530000 (00003000 bytes committed)
4: 00b10000
Segment at 00b10000 to 00b50000 (00001000 bytes committed)
5: 001a0000
Segment at 001a0000 to 001b0000 (00003000 bytes committed)
6: 00170000
Segment at 00170000 to 00180000 (00008000 bytes committed)
7: 013b0000
Segment at 013b0000 to 013c0000 (00003000 bytes committed)
but GC heap and the number of Leak and System.Byte[] objects in it were growing significantly:
Process Uptime: 0 days 0:00:04.000
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x013c1018
generation 1 starts at 0x013c100c
generation 2 starts at 0x013c1000
ephemeral segment allocation context: (0x013cd804, 0x013cdff4)
segment begin allocated size
0055ee08 790d7ae4 790f7064 0x0001f580(128384)
013c0000 013c1000 013cdff4 0x0000cff4(53236)
Large object heap starts at 0x023c1000
segment begin allocated size
023c0000 023c1000 023c3250 0x00002250(8784)
Total Size 0x2e7c4(190404)
------------------------------
GC Heap Size 0×2e7c4(190404)
0:000> !dumpheap -stat
total 2176 objects
Statistics:
Count TotalSize Class Name
...
...
...
46 736 CLRHeapLeak.Leak
5 8816 System.Object[]
46 47656 System.Byte[]
2035 129604 System.String
Total 2176 objects
Process Uptime: 0 days 0:09:56.000
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x018cddbc
generation 1 starts at 0x01541ec4
generation 2 starts at 0x013c1000
ephemeral segment allocation context: (0x0192d668, 0x0192ddc8)
segment begin allocated size
0055ee08 790d7ae4 790f7064 0x0001f580(128384)
013c0000 013c1000 0192ddc8 0x0056cdc8(5688776)
Large object heap starts at 0x023c1000
segment begin allocated size
023c0000 023c1000 023c3240 0x00002240(8768)
Total Size 0x58e588(5825928)
------------------------------
GC Heap Size 0×58e588(5825928)
0:000> !dumpheap -stat
total 12887 objects
Statistics:
Count TotalSize Class Name
...
...
...
5 8816 System.Object[]
5403 86448 CLRHeapLeak.Leak
2026 128632 System.String
5403 5597508 System.Byte[]
Total 12887 objects
Process Uptime: 0 days 0:16:33.000
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x01c59cb4
generation 1 starts at 0x0194fd20
generation 2 starts at 0x013c1000
ephemeral segment allocation context: (0x01cd3050, 0x01cd3cc0)
segment begin allocated size
0055ee08 790d7ae4 790f7064 0x0001f580(128384)
013c0000 013c1000 01cd3cc0 0x00912cc0(9514176)
Large object heap starts at 0x023c1000
segment begin allocated size
023c0000 023c1000 023c3240 0x00002240(8768)
Total Size 0x934480(9651328)
------------------------------
GC Heap Size 0×934480(9651328)
0:000> !dumpheap -stat
total 20164 objects
Statistics:
Count TotalSize Class Name
5 8816 System.Object[]
2026 128632 System.String
9038 144608 CLRHeapLeak.Leak
9038 9363368 System.Byte[]
Total 20164 objects
This is not the traditional memory leak because we have the reference chain. However, uncontrolled memory growth can be considered as a memory leak too, caused by poor application design, bad input validation or error handling, etc.
There are situations when customers think there is a memory leak but it is not. One of them is unusually big size of a process when running it on a multi-processor server. If dllhost.exe hosting typical .NET assembly DLL occupies less than 100Mb on a local workstation starts consuming more than 300Mb on a 4 processor server than it can be the case that the server version of CLR uses per processor GC heaps:
0:000> .loadby sos mscorsvr
0:000> !EEHeap -gc
generation 0 starts at 0×05c80154
generation 1 starts at 0×05c7720c
generation 2 starts at 0×102d0030
generation 0 starts at 0×179a0444
generation 1 starts at 0×1799b7a4
generation 2 starts at 0×142d0030
generation 0 starts at 0×0999ac88
generation 1 starts at 0×09990cc4
generation 2 starts at 0×182d0030
generation 0 starts at 0×242eccb0
generation 1 starts at 0×242d0030
generation 2 starts at 0×1c2d0030
…
…
…
GC Heap Size 0×109702ec(278332140)
or if this is CLR 1.x the old extension will tell you the same too:
0:000> !.\clr10\sos.eeheap -gc
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Number of GC Heaps: 4
------------------------------
Heap 0 (0x000f9af0)
generation 0 starts at 0x05c80154
generation 1 starts at 0x05c7720c
generation 2 starts at 0x102d0030
...
...
...
Heap Size 0x515ed60(85,323,104)
------------------------------
Heap 1 (0x000fa070)
generation 0 starts at 0x179a0444
generation 1 starts at 0x1799b7a4
generation 2 starts at 0x142d0030
...
...
...
Heap Size 0x37c7bf0(58,489,840)
------------------------------
Heap 2 (0x000fab80)
generation 0 starts at 0x0999ac88
generation 1 starts at 0x09990cc4
generation 2 starts at 0x182d0030
...
...
...
Heap Size 0x485de34(75,882,036)
------------------------------
Heap 3 (0x000fb448)
generation 0 starts at 0x242eccb0
generation 1 starts at 0x242d0030
generation 2 starts at 0x1c2d0030
...
...
...
Heap Size 0x41ea570(69,117,296)
------------------------------
Reserved segments:
------------------------------
GC Heap Size 0x1136ecf4(288,812,276)
The more processors you have the more heaps are contributing to the overall VM size. Although the process occupies almost 400Mb if it doesn’t grow constantly over time beyond that value then it is normal.
- Dmitry Vostokov @ DumpAnalysis.org -
Waiting for this book to be released:
Advanced Windows Debugging by Mario Hewardt and Daniel Pravat
Already ordered it and will post my review as soon as it arrives.
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes we suspect that a problem was caused by some module but WinDbg lmv command doesn’t show the company name and other verbose information for it and Google search has no results for the file name. I call this pattern Unknown Component.
In such cases additional information can be obtained by dumping the module resource section or the whole module address range and looking for ASCII and UNICODE strings. For example (byte values in db output are omitted for clarity):
2: kd> lmv m driver
start end module name
f5022000 f503e400 driver (deferred)
Image path: \SystemRoot\System32\drivers\driver.sys
Image name: driver.sys
Timestamp: Tue Jun 12 11:33:16 2007 (466E766C)
CheckSum: 00021A2C
ImageSize: 0001C400
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
2: kd> db f5022000 f503e400
f5022000 MZ..............
f5022010 ........@.......
f5022020 ................
f5022030 ................
f5022040 ........!..L.!Th
f5022050 is program canno
f5022060 t be run in DOS
f5022070 mode....$.......
f5022080 .g,._.B._.B._.B.
f5022090 _.C.=.B..%Q.X.B.
f50220a0 _.B.].B.Y%H.|.B.
f50220b0 ..D.^.B.Rich_.B.
f50220c0 ........PE..L...
f50220d0 lvnF............
...
...
...
f503ce30 ................
f503ce40 ................
f503ce50 ................
f503ce60 ............0...
f503ce70 ................
f503ce80 ....H...........
f503ce90 ..........4...V.
f503cea0 S._.V.E.R.S.I.O.
f503ceb0 N._.I.N.F.O.....
f503cec0 ................
f503ced0 ........?.......
f503cee0 ................
f503cef0 ....P.....S.t.r.
f503cf00 i.n.g.F.i.l.e.I.
f503cf10 n.f.o...,.....0.
f503cf20 4.0.9.0.4.b.0...
f503cf30 4.....C.o.m.p.a.
f503cf40 n.y.N.a.m.e.....
f503cf50 M.y.C.o.m.p. .A.
f503cf60 G...p.$...F.i.l.
f503cf70 e.D.e.s.c.r.i.p.
f503cf80 t.i.o.n.....M.y.
f503cf90 .B.i.g. .P.r.o.
f503cfa0 d.u.c.t. .H.o.o.
f503cfb0 k...............
f503cfc0 ................
f503cfd0 ....4.....F.i.l.
f503cfe0 e.V.e.r.s.i.o.n.
f503cff0 ....5...1...0...
f503d000 ????????????????
f503d010 ????????????????
f503d020 ????????????????
f503d030 ????????????????
...
...
...
We see that CompanyName is MyComp AG, FileDescription is My Big Product Hook and FileVersion is 5.0.1.
In our example the same information can be retrieved by dumping the image file header and then finding and dumping the resource section:
2: kd> lmv m driver
start end module name
f5022000 f503e400 driver (deferred)
Image path: \SystemRoot\System32\drivers\driver.sys
Image name: driver.sys
Timestamp: Tue Jun 12 11:33:16 2007 (466E766C)
CheckSum: 00021A2C
ImageSize: 0001C400
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
2: kd> !dh f5022000 -f
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
14C machine (i386)
6 number of sections
466E766C time date stamp Tue Jun 12 11:33:16 2007
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
10E characteristics
Executable
Line numbers stripped
Symbols stripped
32 bit word machine
OPTIONAL HEADER VALUES
10B magic #
6.00 linker version
190A0 size of code
30A0 size of initialized data
0 size of uninitialized data
1A340 address of entry point
2C0 base of code
----- new -----
00010000 image base
20 section alignment
20 file alignment
1 subsystem (Native)
4.00 operating system version
0.00 image version
4.00 subsystem version
1C400 size of image
2C0 size of headers
21A2C checksum
00100000 size of stack reserve
00001000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
0 [ 0] address [size] of Export Directory
1A580 [ 50] address [size] of Import Directory
1AE40 [ 348] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
1B1A0 [ 1084] address [size] of Base Relocation Directory
420 [ 1C] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
0 [ 0] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
2C0 [ 15C] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory
2: kd> db f5022000+1AE40 f5022000+1AE40+348
f503ce40 ................
f503ce50 ................
f503ce60 ............0...
f503ce70 ................
f503ce80 ....H...........
f503ce90 ..........4...V.
f503cea0 S._.V.E.R.S.I.O.
f503ceb0 N._.I.N.F.O.....
f503cec0 ................
f503ced0 ........?.......
f503cee0 ................
f503cef0 ....P.....S.t.r.
f503cf00 i.n.g.F.i.l.e.I.
f503cf10 n.f.o...,.....0.
f503cf20 4.0.9.0.4.b.0...
f503cf30 4.....C.o.m.p.a.
f503cf40 n.y.N.a.m.e.....
f503cf50 M.y.C.o.m.p. .A.
f503cf60 G...p.$...F.i.l.
f503cf70 e.D.e.s.c.r.i.p.
f503cf80 t.i.o.n.....M.y.
f503cf90 .B.i.g. .P.r.o.
f503cfa0 d.u.c.t. .H.o.o.
f503cfb0 k...............
f503cfc0 ................
f503cfd0 ....4.....F.i.l.
f503cfe0 e.V.e.r.s.i.o.n.
f503cff0 ....5...1...0...
f503d000 ????????????????
f503d010 ????????????????
...
...
...
- Dmitry Vostokov @ DumpAnalysis.org -
This security warning is related to sound files and pictures produced by Dump2Wave, Dump2Picture and WinDbg scripts based on them.
These tools do not change computer memory data and it is present in resulting WAV and BMP files unmodified. Do not publish these files on Internet, otherwise you might expose your private or sensitive data.
If you use lossy compression afterwards, like MP3 or JPEG, all original computer memory content is lost and becomes non-recoverable.
Therefore, if you create a piece of modern art using Dump2Picture and want to publish it electronically always transform it into JPEG, for example.
- Dmitry Vostokov @ DumpAnalysis.org -
An alternative to converting memory dumps to picture files is to save a memory range to a binary file and then convert it to a BMP file. Thus you can view the particular DLL or driver mapped into address space, heap or pool region, etc.
To save a memory range to a file use WinDbg .writemem command:
.writemem d2p-range.bin 00800000 0085e000
or
.writemem d2p-range.bin 00400000 L20000
I wrote a WinDbg script that saves a specified memory range and then calls a shell script which automatically converts saved binary file to a BMP file and then runs whatever picture viewer is registered for .bmp extension.
The WinDbg script code (mempicture.txt):
.writemem d2p-range.bin ${$arg1} ${$arg2}
.if (${/d:$arg3})
{
.shell -i- mempicture.cmd d2p-range ${$arg3}
}
.else
{
.shell -i- mempicture.cmd d2p-range
}
The shell script (mempicture.cmd):
dump2picture %1.bin %1.bmp %2
%1.bmp
Because WinDbg installation folder is assumed to be the default directory for both scripts and Dump2Picture.exe they should be copied to the same folder where windbg.exe is located. On my system it is
C:\Program Files\Debugging Tools for Windows
Both scripts are now included in Dump2Picture package available for free download:
To call the script from WinDbg use the following command:
$$>a< mempicture.txt Range [bits-per-pixel]
where Range can be in Address1 Address2 or Address Lxxx format, bits-per-pixel can be 8, 16, 24 or 32. By default it is 32.
For example, I loaded a complete Windows x64 memory dump and visualized HAL (hardware abstraction layer) module:
kd> lm
start end module name
fffff800`00800000 fffff800`0085e000 hal
fffff800`01000000 fffff800`0147b000 nt
fffff97f`ff000000 fffff97f`ff45d000 win32k
...
...
...
kd> $$>a< mempicture.txt fffff800`00800000 fffff800`0085e000
Writing 5e001 bytes...
C:\Program Files\Debugging Tools for Windows>dump2picture d2p-range.bin d2p-range.bmp
Dump2Picture version 1.1
Written by Dmitry Vostokov, 2007
d2p-range.bmp
d2p-range.bin
1 file(s) copied.
C:\Program Files\Debugging Tools for Windows>d2p-range.bmp
<.shell waiting 10 second(s) for process>
.shell: Process exited
kd>
and Windows Picture and Fax Viewer application was launched and displayed the following picture:
Enjoy 
- Dmitry Vostokov @ DumpAnalysis.org -
I generated a memory dump with plenty of Unicode and ASCII strings “Hello World!” to see how they look on a picture. I assume you know the difference between Unicode (UTF-16) and ASCII encodings: wide characters from the former occupy two bytes:
0:000> db 008c7420 l20
008c7420 48 00 65 00 6c 00 6c 00-6f 00 20 00 57 00 6f 00 H.e.l.l.o. .W.o.
008c7430 72 00 6c 00 64 00 21 00-00 00 00 00 00 00 00 00 r.l.d.!.........
and characters from the latter occupy one byte of memory:
0:000> db 008c72b4 l10
008c72b4 48 65 6c 6c 6f 20 57 6f-72 6c 64 21 00 00 00 00 Hello World!....
You can see that the second byte for Unicode English characters is zero. I converted that memory dump into 8 bits-per-pixel bitmap using Dump2Picture and after zooming it sufficiently in Vista Photo Viewer until pixels become squares I got the following picture that illustrates the difference between Unicode and ASCII strings:
Incidentally the same memory dump converted to 32 bits-per-pixel bitmap shows Unicode “Hello World!” strings in green colors 
- Dmitry Vostokov @ DumpAnalysis.org -
Previously announced Dump2Picture has been updated to version 1.1 with the following improvements to handle 8 bits-per-pixel images correctly:
- Saves grey scale palette
- Calculates right bitmap width and file size
The update can be downloaded from the same link:
Now 8 bits-per-pixel Vista kernel dump looks much better:
- Dmitry Vostokov @ DumpAnalysis.org -
Dump2Picture can be used to explore memory leaks visually. I created the following small program in Visual C++ that leaks 64Kb every second:
#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
while (true)
{
printf("%x\n", (UINT_PTR)malloc(0xFFFF));
Sleep(1000);
}
return 0;
}
Then I sampled 3 dumps at 7Mb, 17Mb and 32Mb process virtual memory size and converted them as 16 bits-per-pixel bitmaps. On the pictures below we can see that the middle black memory area grows significantly. Obviously malloc function allocates zeroed memory and therefore we see black color.
7Mb process memory dump:
17Mb process memory dump:
32Mb process memory dump:
If we zoom in the black area we would see the following pattern:
Colored lines inside are heap control structures that are created for every allocated block of memory. If this is correct then allocating smaller memory blocks would create a hatched pattern. And this is true indeed. The following program leaks 256 byte memory blocks:
#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
while (true)
{
printf("%x\n", (UINT_PTR)malloc(0xFF));
Sleep(1000/0xFF);
}
return 0;
}
The corresponding process memory picture and zoomed heap area are the following:
Making allocations 4 times smaller makes heap area to look hatched and zoomed picture is more densely packed by heap control structures:
#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
while (true)
{
printf("%x\n", (UINT_PTR)malloc(0xFF/4));
Sleep((1000/0xFF)/4);
}
return 0;
}
Here is another example. One service was increasing its memory constantly. The crash dump picture shows huge hatched dark region in the middle:
and if we zoom in this region:
Because the pattern and allocation size look uniform it could be the true heap memory leak for some operation that allocates constant size buffers. After opening the dump and looking at heap segments that had grown the most we see the same allocation size indeed:
0:000> !.\w2kfre\ntsdexts.heap -h 5
HEAPEXT: Unable to get address of NTDLL!NtGlobalFlag.
Index Address Name Debugging options enabled
1: 00140000
2: 00240000
3: 00310000
4: 00330000
5: 00370000
Segment at 00370000 to 00380000 (00010000 bytes committed)
Segment at 01680000 to 01780000 (00100000 bytes committed)
Segment at 019C0000 to 01BC0000 (00200000 bytes committed)
Segment at 01BC0000 to 01FC0000 (00400000 bytes committed)
Segment at 01FC0000 to 027C0000 (00800000 bytes committed)
Segment at 027C0000 to 037C0000 (01000000 bytes committed)
Segment at 037C0000 to 057C0000 (02000000 bytes committed)
Segment at 057C0000 to 097C0000 (00155000 bytes committed)
…
…
…
057B96E0: 01048 . 01048 [07] - busy (1030), tail fill
057BA728: 01048 . 01048 [07] - busy (1030), tail fill
057BB770: 01048 . 01048 [07] - busy (1030), tail fill
057BC7B8: 01048 . 01048 [07] - busy (1030), tail fill
057BD800: 01048 . 01048 [07] - busy (1030), tail fill
057BE848: 01048 . 01048 [07] - busy (1030), tail fill
057BF890: 01048 . 00770 [14] free fill
Heap entries for Segment07 in Heap 370000
057C0040: 00040 . 01048 [07] - busy (1030), tail fill
057C1088: 01048 . 01048 [07] - busy (1030), tail fill
057C20D0: 01048 . 01048 [07] - busy (1030), tail fill
057C3118: 01048 . 01048 [07] - busy (1030), tail fill
057C4160: 01048 . 01048 [07] - busy (1030), tail fill
057C51A8: 01048 . 01048 [07] - busy (1030), tail fill
…
…
…
- Dmitry Vostokov @ DumpAnalysis.org -
