Archive for the ‘WinDbg Scripts’ Category

The Korean Edition of Memory Dump Analysis Anthology, Volume 1

Monday, April 5th, 2010

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Memory Dump Analysis Anthology, Volume 4

Thursday, February 11th, 2010

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 4
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 410 pages
  • Publisher: Opentask (30 March 2010)
  • ISBN-13: 978-1-906717-86-5
  • Hardcover: 410 pages
  • Publisher: Opentask (30 April 2010)
  • ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Complete Stack Traces from x64 System

Tuesday, February 9th, 2010

Previously I wrote on how to get a 32-bit stack trace from a 32-bit process thread on an x64 system. There are situations when we are interested in all such stack traces, for example, from a complete memory dump. I wrote a script that extracts both 64-bit and WOW64 32-bit stack traces:

.load wow64exts
!for_each_thread "!thread @#Thread 1f;.thread /w @#Thread; .reload; kb 256; .effmach AMD64"

Example output fragment for a thread fffffa801f3a3bb0 from a very long debugger log file:

[...]

Setting context for owner process...
.process /p /r fffffa8013177c10

                                    

THREAD fffffa801f3a3bb0  Cid 4b4c.5fec  Teb: 000000007efaa000 Win32Thread: fffff900c1efad50 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8021ce4590  NotificationEvent
    fffffa801f3a3c68  NotificationTimer
Not impersonating
DeviceMap                 fffff8801b551720
Owning Process            fffffa8013177c10       Image:         application.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      14066428       Ticks: 301 (0:00:00:04.695)
Context Switch Count      248                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address mscorwks!Thread::intermediateThreadProc (0x00000000733853b3)
Stack Init fffffa60190e5db0 Current fffffa60190e5940
Base fffffa60190e6000 Limit fffffa60190df000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`190e5980 fffff800`01cba0fa nt!KiSwapContext+0x7f
fffffa60`190e5ac0 fffff800`01caedab nt!KiSwapThread+0x13a
fffffa60`190e5b30 fffff800`01f1d608 nt!KeWaitForSingleObject+0x2cb
fffffa60`190e5bc0 fffff800`01cb7973 nt!NtWaitForSingleObject+0x98
fffffa60`190e5c20 00000000`75183d09 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffffa60`190e5c20)
00000000`069ef118 00000000`75183b06 wow64cpu!CpupSyscallStub+0x9
00000000`069ef120 00000000`74f8ab46 wow64cpu!Thunk0ArgReloadState+0x1a
00000000`069ef190 00000000`74f8a14c wow64!RunCpuSimulation+0xa
00000000`069ef1c0 00000000`771605a8 wow64!Wow64LdrpInitialize+0x4b4
00000000`069ef720 00000000`771168de ntdll! ?? ::FNODOBFM::`string'+0x20aa1
00000000`069ef7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

.process /p /r 0
Implicit thread is now fffffa80`1f3a3bb0
WARNING: WOW context retrieval requires
switching to the thread's process context.
Use .process /p fffffa80`1f6b2990 to switch back.
Implicit process is now fffffa80`13177c10
x86 context set
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols
ChildEBP RetAddr
06aefc68 76921270 ntdll_772b0000!ZwWaitForSingleObject+0x15
06aefcd8 7328c639 kernel32!WaitForSingleObjectEx+0xbe
06aefd1c 7328c56f mscorwks!PEImage::LoadImage+0x1af
06aefd6c 7328c58e mscorwks!CLREvent::WaitEx+0x117
06aefd80 733770fb mscorwks!CLREvent::Wait+0x17
06aefe00 73377589 mscorwks!ThreadpoolMgr::SafeWait+0x73
06aefe64 733853f9 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11c
06aeff88 7699eccb mscorwks!Thread::intermediateThreadProc+0x49
06aeff94 7732d24d kernel32!BaseThreadInitThunk+0xe
06aeffd4 7732d45f ntdll_772b0000!__RtlUserThreadStart+0x23
06aeffec 00000000 ntdll_772b0000!_RtlUserThreadStart+0x1b
Effective machine: x64 (AMD64)

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Native Script Debugging

Monday, February 1st, 2010

While testing a WinDbg script for the CARE system (the script enumerates files on a Windows PC and processes memory dumps to generate a log file with the output of debugger commands) I found that after successful processing of many files the next launched WinDbg instance suddenly showed this message box:

To find out, I attached another WinDbg instance to its process in order to examine the real command line. In this small case study instead of using kb WinDbg command to show a stack trace and its arguments I employ kn, .frame and kb <lines> commands for visual clarity and to illustrate strack trace frame navigation. In the failed WinDbg instance that had just started we see only one thread showing Message Box pattern:

0:000> ~*kn

.  0  Id: dc8.fb4 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`0025d4b8 00000000`76fc5118 USER32!NtUserWaitMessage+0xa
01 00000000`0025d4c0 00000000`76fc5770 USER32!DialogBox2+0x261
02 00000000`0025d540 00000000`7701000d USER32!InternalDialogBox+0x134
03 00000000`0025d5a0 00000000`7700f2b8 USER32!SoftModalMessageBox+0x9fb
04 00000000`0025d6d0 00000000`7700eb17 USER32!MessageBoxWorker+0x314
05 00000000`0025d890 00000000`7700ea10 USER32!MessageBoxTimeoutW+0xb3
06 00000000`0025d950 00000001`3f9016a6 USER32!MessageBoxW+0x4c
07 00000000`0025d990 00000001`3f90175c WinDbg!TextMsgBox+0x96
08 00000000`0025d9d0 00000001`3f9017d7 WinDbg!FormatMsgBoxV+0x9c
09 00000000`0025dbe0 00000001`3f9075c7 WinDbg!InfoBox+0x37
0a 00000000`0025dc20 00000001`3f9084f7 WinDbg!ParseCommandLine+0x1a57
0b 00000000`0025dcc0 00000001`3f913739 WinDbg!wmain+0×287
0c 00000000`0025fd80 00000000`7708be3d WinDbg!_CxxFrameHandler3+0×291
0d 00000000`0025fdc0 00000000`771c6a51 kernel32!BaseThreadInitThunk+0xd
0e 00000000`0025fdf0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see the frame # 0b contains the return address of wmain function (starting point of execution of UNICODE C/C++ programs) that has this prototype:

int wmain(int argc, wchar_t *argv[], wchar_t *envp[]);

We switch to that frame to examine its first 3 parameters and use kb command that shows stack traces starting from the current frame (we are interested in the top stack trace line only):

0:000> .frame b
0b 00000000`0025dcc0 00000001`3f913739 WinDbg!wmain+0×287

0:000> kb 1
RetAddr           : Args to Child                                                           : Call Site
00000001`3f913739 : 00000000`0000000c 00000000`00278b60 00000000`00279e10 000007de`a4ecc920 : WinDbg!wmain+0×287

Because the function prototype shows the second function parameter as an array of wide character null-terminated strings we use dpu command to dump them. We also note that we have only 0xc array members and use this as the length argument for dpu:

0:000> dpu 00000000`00278b60 Lc
00000000`00278b60  00000000`00278bc8 “C:\Program Files\Debugging Tools for Windows (x64)\WinD”
00000000`00278b68  00000000`00278c44 “-y”
00000000`00278b70  00000000`00278c4a “srv*c:\ms*http://msdl.microsoft.com/download/symbols;sr”
00000000`00278b78  00000000`00278d0c “-z”
00000000`00278b80  00000000`00278d12 “C:\MemoryDumps\CST\ColorimetricTracing”
00000000`00278b88  00000000`00278d60 “(4).DMP”

00000000`00278b90  00000000`00278d70 “-c”
00000000`00278b98  00000000`00278d76 “$$>a<DebuggerLogs.txt;q”
00000000`00278ba0  00000000`00278da6 “-Q”
00000000`00278ba8  00000000`00278dac “-QS”
00000000`00278bb0  00000000`00278db4 “-QY”
00000000`00278bb8  00000000`00278dbc “-QSY”

We see in the output above that “C:\MemoryDumps\CST\ColorimetricTracing” and “(4).DMP” strings were probably split from one file name “C:\MemoryDumps\CST\ColorimetricTracing (4).DMP” and this means that we forgot to enclose the file name parameter in double quotation marks when passing it from VB script to WinDbg.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Mode-independent WinDbg scripts

Friday, January 29th, 2010

These are scripts that can be run without modification in both user and kernel modes to collect information from user and kernel spaces. For example, we want to collect thread stack traces for CARE system and we have different kinds of memory dumps stored on our computer. There is no a single command that can show stack traces for all threads in a process and kernel / complete memory dumps. However, we can combine separate mode-sensitive commands in one script:

.kframes 1000
!for_each_thread !thread @#Thread 1f
~*kv

The first command eliminates the common mistake of truncated traces. The second command fails for process user memory dumps but shows full 3-parameter stack trace for every thread in a system including user space thread stack counterpart for complete memory dumps after switching to the appropriate process context if any. The third command fails for kernel and complete memory dumps but lists stack traces for each thread in a process user memory dump. Therefore, we have just one script that we can run against all memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -  

Memory Dump Analysis Anthology, Volume 3

Sunday, December 20th, 2009

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Forthcoming Memory Dump Analysis Anthology, Volume 3

Saturday, September 26th, 2009

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 3
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 404 pages
  • Publisher: Opentask (20 December 2009)
  • ISBN-13: 978-1-906717-43-8
  • Hardcover: 404 pages
  • Publisher: Opentask (30 January 2010)
  • ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

Pattern-Driven Memory Analysis (Part 3)

Thursday, May 14th, 2009

Part 2 briefly discussed debuggers and their commands. Debugger commands can be grouped together into scripts that can be run against memory dump files and their resulted textual output can be redirected to log files.

Therefore we start with the first step in our analysis process diagram introduced in Part 1:

Information Collection: Scripts

Here a script is a sequence of instructions to extract formatted textual information from a memory dump file (or a live system). There are many advantages of scripts and logs. Many companies, for example, banks, avoid sending plain memory dump files because of security considerations in order to prevent exposure of company or private information. After the advent of 64-bit personal computing complete memory dump files became larger and larger and it is now common to get 32Gb memory dumps. Processing multiple long-running commands can easily be done from textual log files. Additionally, scripts can be used to process hundreds of memory dumps in one go to look for similarities and differences. Many tools can be used for such purpose including built-in debugger scripting capabilities, shell scripting and regular expressions for intelligent search.

- Dmitry Vostokov @ DumpAnalysis.org -

Raw Stack Dump of all threads (part 3)

Monday, May 11th, 2009

Sometimes the script featured in part 1 doesn’t work because of the lack of symbols or something else:

***                                                                  
***                                                                  
*** Your debugger is not using the correct symbols    
*** 
*** In order for this command to work properly, your symbol path
*** must point to .pdb files that have full type information.
*** 
*** Certain .pdb files (such as the public OS symbols) do not
*** contain the required information.  Contact the group that
*** provided you with these symbols if you need this command to
*** work.
*** 
*** Type referenced: ntdll!_NT_TIB
***
Couldn’t resolve error at ‘ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2′
                    ^ Extra character error in ‘~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2′

This is the case where !teb WinDbg command doesn’t work and we can cope with this as shown in the following post:

Coping with missing symbolic information

Therefore we can adjust our user-mode script to use hard-coded offsets and delineate raw stack outputs by the output of kv WinDbg command:

~*e r? $t0 = @$teb; r? $t1 = @$t0+8; r? $t2 = @$t0+4; kv 100; dps poi(@$t1) poi(@$t2)

- Dmitry Vostokov @ DumpAnalysis.org -

A Windows case for delta debugging

Thursday, May 7th, 2009

My local browser crashed today when I did copy-paste of an RTF text into an HTML editor window. The dump was not saved because I previously set up logging as described here (my script doesn’t include .dump commands):

All at once: postmortem logs and dump files

Looking at stack trace I noticed that the crash happened during HTML processing (call arguments are removed for visual clarity):

STACK_TEXT: 
0476de3c 6970d597 html!FPseudoStyleBis+0x26
0476de48 69703b0e html!BisFromLpxszStyle+0x1c
0476de60 69702ba9 html!LwMultDivRU+0x4b6
0476dea4 6970518a html!FMarkListCallback+0x56c
0476deb4 697068b7 html!JcCalcFromXaExtents+0x91
0476df60 697070c5 html!EmitNonBreakingSpace+0x445
0476e08c 697107ff html!FEmitHtmlFnOtag+0x17d
0476e0b0 696ec6a8 html!ConvertRtfToForeign+0x105
0476e538 696ec745 html!FceRtfToForeign+0x266
0476e560 6b7e5ad4 html!RtfToForeign32+0x51
0476e9a8 6b7e5c83 mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml+0x163
0476edfc 6b79cc15 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x11a
0476ee18 6b79cd81 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x38
0476ee2c 6b7cdcea mshtmled!CHTMLEditor::ConvertRTFToHTML+0x12
0476ee98 6b7ce392 mshtmled!CPasteCommand::PasteFromClipboard+0x2c0
0476ef08 6b78d218 mshtmled!CPasteCommand::PrivateExec+0x47a
0476ef2c 6b78d1ad mshtmled!CCommand::Exec+0x4b
0476ef50 6b470d14 mshtmled!CMshtmlEd::Exec+0xf9
0476ef80 6b4688a8 mshtml!CEditRouter::ExecEditCommand+0xd6
0476f328 6b5ceccf mshtml!CDoc::ExecHelper+0x338d
0476f374 6b468a2f mshtml!CFrameSite::Exec+0x264
0476f3a8 6b4687af mshtml!CDoc::RouteCTElement+0xf1
0476f740 6b468586 mshtml!CDoc::ExecHelper+0x325e
0476f760 6b510e7b mshtml!CDoc::Exec+0x1e
0476f798 6b48a708 mshtml!CDoc::OnCommand+0x9c
0476f8ac 6b3997e1 mshtml!CDoc::OnWindowMessage+0x841
0476f8d8 766ff8d2 mshtml!CServer::WndProc+0x78
0476f904 766ff794 USER32!InternalCallWinProc+0x23
0476f97c 767006f6 USER32!UserCallWinProcCheckWow+0x14b
0476f9ac 7670069c USER32!CallWindowProcAorW+0x97
0476f9cc 6ce1851b USER32!CallWindowProcW+0x1b
WARNING: Stack unwind information not available. Following frames may be wrong.
0476fa40 6ce0cdc6 GoogleToolbarDynamic_6D0D6FD66D664927!DllGetClassObject+0x24981
0476fa64 6ce9beaa GoogleToolbarDynamic_6D0D6FD66D664927!DllGetClassObject+0x1922c
0476fa94 766ff8d2 GoogleToolbarDynamic_6D0D6FD66D664927!DllGetClassObject+0xa8310
0476fac0 766ff794 USER32!InternalCallWinProc+0x23
0476fb38 76700a05 USER32!UserCallWinProcCheckWow+0x14b
0476fb78 76700afa USER32!SendMessageWorker+0x4b7
0476fb98 6b47fb9b USER32!SendMessageW+0x7c
0476fbc4 6b3d8e5a mshtml!CElement::PerformTA+0x71
0476fbe4 6b3d8db9 mshtml!CDoc::PerformTA+0xd8
0476fc60 6b46381c mshtml!CDoc::PumpMessage+0x8e0
0476fd14 6b463684 mshtml!CDoc::DoTranslateAccelerator+0x33f
0476fd30 6b4634cc mshtml!CServer::TranslateAcceleratorW+0x56
0476fd50 70c9f550 mshtml!CDoc::TranslateAcceleratorW+0x83
0476fd6c 70c9f600 IEFRAME!CProxyActiveObject::TranslateAcceleratorW+0x30
0476fd90 70c9fca1 IEFRAME!CDocObjectView::TranslateAcceleratorW+0xb1
0476fdb0 70c9faf4 IEFRAME!CCommonBrowser::v_MayTranslateAccelerator+0xda
0476fddc 70c9f7b0 IEFRAME!CShellBrowser2::_MayTranslateAccelerator+0x68
0476fdec 70c9f7f5 IEFRAME!CShellBrowser2::v_MayTranslateAccelerator+0x15
0476fe58 76894911 IEFRAME!CTabWindow::_TabWindowThreadProc+0x264
0476fe64 776ae4b6 kernel32!BaseThreadInitThunk+0xe
0476fea4 776ae489 ntdll!__RtlUserThreadStart+0x23
0476febc 00000000 ntdll!_RtlUserThreadStart+0x1b

I immediately recalled that in Andreas Zeller’s book Why Programs Fail a browser parsing HTML was used as an example to show delta debugging (I think it was Mozilla).

The complete log file can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org -

Pattern-Driven Memory Analysis (Part 2)

Tuesday, April 21st, 2009

Before we explain stages of the analysis process shown in Part 1, let’s start with a brief overview of memory dumps, debuggers and logs. Recall that a memory dump is a snapshot of a process, system or physical memory state. This unifies post-mortem analysis and live debugging. Debuggers are tools that allow us to get and modify these memory snapshots. Other tools that allow us to get memory dump files are process dumpers like userdump.exe, Task Manager since Vista, WER, and system dumpers like LiveKd and Win32dd. We should not forget tools and methods that allow us to trigger Windows kernel ability to save consistent memory dump files: NMI button, keyboard method and various software bugcheck-triggers like Citrix SystemDump. Now coming back to debuggers. One of their essential features is to save a debugging session log, formatted textual output saved in a text file for further processing. One good example is !process 0 ff WinDbg command to output all processes and their thread stack traces (see Stack Trace Collection pattern for other variations). 

I’ve created a page to add all P-DMA parts as soon as I write them:

Pattern-Driven Memory Analysis

- Dmitry Vostokov @ DumpAnalysis.org -

March issue of Debugged! MZ/PE is available!

Sunday, March 29th, 2009

Finally it has been published and available for orders from Amazon and other bookstores:

http://www.dumpanalysis.org/Debugged+Magazine

I had to increase the number of pages for the first issue from 16, planned originally, to 28 and this is reflected in the retail price of $10 (originally planned $8) but bookstores should sell it with a discount between 0% and 55%.

More information about the next issue should be ready by the end of the next week.

- Dmitry Vostokov @ DumpAnalysis.org

Debugged! MZ/PE soon to be available!

Wednesday, March 18th, 2009

Yesterday I submitted the magazine to print and distribution world-wide. If everything is right it should be available by the end of this month. This first issue features 12 page WinDbg command supplement to pattern-driven memory dump analysis methodology, an overview of Win32dd complete memory dumper and PowerDbg enhancements to debug ASP.NET code. The magazine will only be available in print.

- Dmitry Vostokov @ DumpAnalysis.org -

Pattern-Driven Memory Analysis (Part 1)

Monday, March 2nd, 2009

Last week I had an opportunity to present a pattern-driven memory dump analysis methodology at a global engineering conference. Now in a series of articles I’m going to clarify certain points and extend it to wider domain of memory analysis including computer memory forensics and intelligence.

Today I post the reworked picture of a waterfall-like analysis process:

 

Various phases and their relationship will be discussed in subsequent parts together with examples. 

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg In Use: Debugging Exercises

Wednesday, December 24th, 2008

The analogy between learning a complex tool with its own language and a foreign natural language has been developed further after the release of WinDbg Learning Cards and finally culminated in “WinDbg In Use” book series with the first book to be published during the 1st quarter of 2009:

  • Title: WinDbg In Use: Debugging Exercises (Elementary and Intermediate Level)
  • Author: Dmitry Vostokov
  • Publisher: Opentask (15 March 2009)
  • Language: English
  • Product Dimensions: 23.5 x 19.1
  • ISBN-13: 978-1-906717-50-6
  • Paperback: 200 pages
  • Book Annotation: Includes 60 programmed exercises from real life debugging and crash dump analysis scenarios and multiple-choice questions with full answers, comments and suggestions for further reading.

Some example exercises will be published on this blog from time to time. I also plan a corresponding column in the forthcoming Debugged! magazine. 

- Dmitry Vostokov @ DumpAnalysis.org -

Debugged! Magazine

Tuesday, November 25th, 2008

As one of the new initiatives for the Year of Debugging  DumpAnalysis Portal will publish bimonthly full color 16 page publication called:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The only serial publication dedicated entirely to Windows® debugging

The first issue is planned for March, 2009 and will have ISBN-13: 978-1-906717-38-4. If it goes well I’m planning to have ISSN number assigned to it too. More details will be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org

WinDbg: A Reference Poster and Learning Cards

Monday, November 10th, 2008

Suddenly the course of my publishing activities bended a little to produce a DIY poster and learning cards to be published soon. Here are the product details:

Annotation:
WinDbg is a powerful debugger from Microsoft Debugging Tools for Windows. It has more than 350 commands that can be used in different debugging scenarios. The cover of this book is a poster featuring crash dump analysis checklist and common patterns seen in memory dumps and live debugging sessions. Inside the book you can find ready to cut learning cards with commands and their descriptions coloured according to their use for crash dump or live debugging sessions and user, kernel or complete memory dumps. Tossing cards can create unexpected connections between commands and help to learn them more quickly. Uncut pages can also serve as birds eye view to WinDbg debugging capabilities. More than 350 WinDbg commands including meta-commands and extensions are included.

  • Title: WinDbg: A Reference Poster and Learning Cards
  • Authors: Dmitry Vostokov
  • Publisher: Opentask (20 November 2008)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • ISBN-13: 978-1-906717-29-2
  • Paperback: 20 pages

Book Excerpt

Front cover:

Back cover:

After you take inside pages out you are left with a cover that you can use as a crash dump analysis checklist and patterns poster:

I also plan to update this book on a yearly basis. 

- Dmitry Vostokov @ DumpAnalysis.org -

Sparse complete x64 memory dumps

Thursday, October 30th, 2008

Because of the larger virtual address space x64 Windows servers are usually equipped with 16Gb or more physical memory to take advantage of new vast memory layout where pools are “virtually” unlimited and their size is measured in Gb than in Mb (highlighted in enlarged blue font below):

0: kd> !vm

*** Virtual Memory Usage ***
        Physical Memory:     4193970 (  16775880 Kb)
        Page File: \??\C:\pagefile.sys
          Current:  17825792 Kb  Free Space:  17810140 Kb
          Minimum:  17825792 Kb  Maximum:     17825792 Kb
        Page File: \??\D:\pagefile.sys
          Current:  32768000 Kb  Free Space:  32754984 Kb
          Minimum:  32768000 Kb  Maximum:     32768000 Kb
        Available Pages:     3851036 (  15404144 Kb)
        ResAvail Pages:      3951755 (  15807020 Kb)
        Locked IO Pages:         136 (       544 Kb)
        Free System PTEs:   16752738 (  67010952 Kb)
        Free NP PTEs:        1635326 (   6541304 Kb)
        Free Special NP:           0 (         0 Kb)
        Modified Pages:           52 (       208 Kb)
        Modified PF Pages:        38 (       152 Kb)
        NonPagedPool Usage:    12421 (     49684 Kb)
        NonPagedPool Max:    1668607 (   6674428 Kb)
        PagedPool 0 Usage:      9501 (     38004 Kb)
        PagedPool 1 Usage:       604 (      2416 Kb)
        PagedPool 2 Usage:       616 (      2464 Kb)
        PagedPool 3 Usage:       598 (      2392 Kb)
        PagedPool 4 Usage:       603 (      2412 Kb)
        PagedPool Usage:       11922 (     47688 Kb)
        PagedPool Maximum:   6674432 (  26697728 Kb)
        Shared Commit:          2649 (     10596 Kb)
        Special Pool:              0 (         0 Kb)
        Shared Process:         8472 (     33888 Kb)
        PagedPool Commit:      11949 (     47796 Kb)
        Driver Commit:          2603 (     10412 Kb)
        Committed pages:      159687 (    638748 Kb)
        Commit limit:       16686113 (  66744452 Kb)

[...]

It came to my attention today that complete memory dumps can be smaller, sparser in such big memory layouts with many unused physical memory regions. Therefore, complete memory dumps could be smaller than the actual amount of physical memory and even when possibly truncated with many OS structures being included. For the virtual memory stats above the size of complete memory dump was 5Gb and although WinDbg reports the dump as truncated with 16Gb of physical memory it was possible that everything was fit into the first 5Gb of physical memory and saved accordingly in 17Gb page file. For example, !locks command works perfectly (it frequently unable to traverse truncated complete dumps from 32-bit Windows):

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...

Resource @ nt!CmpRegistryLock (0xfffff800011de220)    Shared 1 owning threads
    Contention Count = 11
     Threads: fffffade708e17a0-01<*>
KD: Scanning for held locks...

Resource @ 0xfffffade6f8b1a40    Shared 1 owning threads
     Threads: fffffade708e17a0-01<*>
KD: Scanning for held locks...

6213 total locks, 2 locks currently held

At the same time some data is missing from the file so it could be really truncated dump. For example, the information about computer name is missing:

0: kd> dq srv!srvcomputername l2
fffffade`57919a10  00000000`00220010 fffffa80`01cfa980

0: kd> !address fffffade`57919a10
  fffffade55e04000 - 0000000005ffb000           ffade6e1108e0
          Usage       KernelSpaceUsageNonPagedSystem

0: kd> !pte fffffade`57919a10
                                 VA fffffade57919a10
PXE @ FFFFF6FB7DBEDFA8     PPE at FFFFF6FB7DBF5BC8    PDE at FFFFF6FB7EB795E0    PTE at FFFFF6FD6F2BC8C8
contains 0000000114E00863  contains 000000011CD63863  contains 000000011CE20963  contains 80000000A8265963
pfn 114e00 —DA–KWEV        pfn 11cd63 —DA–KWEV        pfn 11ce20 -G-DA–KWEV        pfn a8265 -G-DA–KW-V

0: kd> du fffffa80`01cfa980 l10
fffffa80`01cfa980  “????????????????”

0: kd> !address fffffa80`01cfa980
  fffffa8000000000 - 000000065d800000           ffade6e1108e0
          Usage       KernelSpaceUsagePagedPool

0: kd> !pte fffffa80`01cfa980
                                 VA fffffa8001cfa980
PXE @ FFFFF6FB7DBEDFA8     PPE at FFFFF6FB7DBF5000    PDE at FFFFF6FB7EA00070    PTE at FFFFF6FD4000E7D0
Unable to get PDE FFFFF6FB7EA00070

Fortunately I got the computer name from a PEB of a randomly selected process though:

0: kd> .process /r /p fffffade6ddd9c20
Implicit process is now fffffade`6ddd9c20
Loading User Symbols
...

0: kd> !peb
PEB at 000000007efdf000
[...]
        COMPUTERNAME=SERVER_A
[…]

I remember that during my Florida trip almost 5 years ago people were worrying about troubleshooting crashes and hangs on 64-bit Windows and discussed how they would send zipped complete memory dumps on several DVD via a courier post. Now with Blu-ray discs (BD) becoming a commodity the size of complete memory dumps is not perceived as a big problem… For really huge dumps WinDbg scripts collecting data on-site might be a solution too (see Dmp2Txt: Solving Security Problem for WinDbg script usage).

- Dmitry Vostokov @ DumpAnalysis.org -

MDAA Volume 2 is available on Amazon and B&N

Saturday, October 18th, 2008

Paperback edition of Memory Dump Analysis Anthology, Volume 2 is finally available on Amazon and Barnes & Noble. Search Inside is also available on Amazon. In addition, I updated the list of recommended books:

Listmania! Crash Dump Analysis and Debugging

Hardcover edition will be available on Amazon and B&N in 2-3 weeks.

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump Analysis Anthology, Volume 2

Friday, October 3rd, 2008

“Everything is memory dump.”

I’m very excited to announce that Volume 2 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 2

In one or two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available by the end of October.

I’m often asked when Volume 3 is available and I currently plan to release it in October - November, 2009. In the mean time I’m planning to concentrate on other publishing projects. 

- Dmitry Vostokov @ DumpAnalysis.org -