Crash Dump Analysis

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free webinar

Date: 18th of August 2010
Time: 21:00 (BST) 16:00 (Eastern) 13:00 (Pacific)
Duration: 90 minutes

Topics include:

- User vs. kernel vs. physical (complete) memory space
- Challenges of complete memory dump analysis
- Common WinDbg commands
- Patterns
- Common mistakes
- Fiber bundles
- Hands-on exercise: a complete memory dump analysis
- A guide to DumpAnalysis.org case studies

Prerequisites: working knowledge of basic user process and kernel memory dump analysis or live debugging using WinDbg 

The webinar link will be posted before 18th of August on DumpAnalysis.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Our future sponsor has been registered in Ireland and has its own independent website and logo: DumpAnalysis.com

More information will be available later this month.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):

• Are you interested in on-site training, prefer traveling or attending webinars?
• Are you interested in software trace analysis as well?
• What specific topics are you interested in?
• What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)

Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.

Looking forward to your responses. Any suggestions are welcome.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As soon as I dug out the 3rd edition of this book to download samples for my internal projects I found that the new edition was published this month! I read all of them and now ordering the 4th edition:

Windows System Programming (4th Edition)

Actually I re-read the 2nd edition of Johnson M. Hart’s book when looking for a job in 2003 and coupled with timely reading of John Robbings’ book Debugging Applications (1st 2000 edition) secured my landing in Dublin East Point Business Park.

This book is an essential reading for Windows memory dump analysts, software maintenance and escalation engineers, software defect researchers and software tool developers. It lucidly describes and succinctly illustrates user-land Windows API with practical console mode samples in plain C. This book is especially valuable for software engineers coming from UNIX background because the author draws various parallels and provides maps between UNIX / Pthreads and Win32 / 64 APIs. Highly recommended! Plan to post an Amazon review when I get the copy of the 4th edition.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 404 pages
• Publisher: Opentask (20 December 2009)
• ISBN-13: 978-1-906717-43-8

• Hardcover: 404 pages
• Publisher: Opentask (30 January 2010)
• ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org accepts hardware such as laptops for reviewing in relation to their suitability for extreme debugging, virtualization, trace analysis, computer forensics, memory dump analysis, visualization and auralization. If you work for a H/W company like HP, Apple, Dell, Acer, Sony or any other respectable manufacturer please don’t hesitate to forward this post to your management: it could be your company brand or laptop model that debugging and software technical support community chooses next time of upgrade or for T&D / R&D! H/W reviews will be posted on the main portal page which currently has an audience of more than 200,000 unique visitors per year from more than 30,000 network locations (*).

If your company is interested please don’t hesitate to use this contact form:

http://www.dumpanalysis.org/contact

(*) From Google Analytics report.

- Dmitry Vostokov @ DumpAnalysis.org -

I’m very pleased to announce the free online version of Debugged! MZ/PE magazine under the code name DEMO launched last night:

Debugging Expert Magazine Online (www.DebuggingExpert.com)

- Dmitry Vostokov @ DumpAnalysis.org -

The digital version of the book is finally available:

x64 Windows Debugging: Practical Foundations

Paperback should be available in 1-2 weeks on Amazon and other stores. When working on the book I fixed errors in the previous x86 version. Errata file for it should be available tomorrow.

- Dmitry Vostokov @ DumpAnalysis.org -

I resumed this week my reading notebook on Software Generalist blog with a top priority book to read every working day: Windows Internals, 5th edition. In reading notes I put what I find interesting for me (at this time) or related to Windows memory dump analysis or debugging and troubleshooting in general. For the latter case, sometimes I put additional references or even WinDbg examples from user, kernel and complete memory dumps in full color. Hope you find these notes useful too:

http://www.softwaregeneralist.com/category/notes-on-windows-internals/

- Dmitry Vostokov @ DumpAnalysis.org -

Got it in the post yesterday shipped via courier from Amazon:

Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (PRO-Developer)

I originally expected it to be a paperback book but as a publisher myself I welcome the final MS Press decision to make it a hardcover. It is more solid. I plan to read it from cover to cover again, starting from tomorrow. I would like to thank authors for including the reference to www.dumpanalysis.org in the conclusion part of the last chapter 14 dedicated to crash dump analysis. This came as a nice surprise for me and I’m going to install a special plaque in the office and put its picture later on my blog.

- Dmitry Vostokov @ DumpAnalysis.org -

I looked at thread raw stack in a process dump from Windows Server 2008 and found ntdll!FinalExceptionHandler symbol:

Loading Dump File [App.dmp]
User Mini Dump File with Full Memory: Only application data is available
Windows Server 2008/Windows Vista Version 6001 (Service Pack 1) MP (4 procs) Free x86 compatible

0:000> dds 00225000 00230000
[...]
0022ff7c  00000000
0022ff80  00000000
0022ff84  00000000
0022ff88  0022ff94
0022ff8c  76744911 kernel32!BaseThreadInitThunk+0xe
0022ff90  7ffdf000
0022ff94  0022ffd4
0022ff98  77b5e4b6 ntdll!__RtlUserThreadStart+0x23
0022ff9c  7ffdf000
0022ffa0  2497b80a
0022ffa4  00000000
0022ffa8  00000000
0022ffac  7ffdf000
0022ffb0  00000000
0022ffb4  00000000
0022ffb8  00000000
0022ffbc  0022ffa0
0022ffc0  00000000
0022ffc4  0022ffe4
0022ffc8  77b29834 ntdll!_except_handler4
0022ffcc  530d7826
0022ffd0  00000000
0022ffd4  0022ffec
0022ffd8  77b5e489 ntdll!_RtlUserThreadStart+0x1b
0022ffdc  00401110 App+0x1110
0022ffe0  7ffdf000
0022ffe4  ffffffff
0022ffe8  77bc75de ntdll!FinalExceptionHandler
0022ffec  00000000
0022fff0  00000000
0022fff4  00401110 App+0×1110
0022fff8  7ffdf000
0022fffc  00000000
00230000  78746341

Search on Internet led me to this very interesting review article about Windows memory protection mechanisms:

http://taossa.com/archive/bh08sotirovdowd.pdf

It also explains this new “Final” exception mechanism in W2K8.

I couldn’t find this in raw stack traces on Windows 7 so it might be the case that it is not enabled by default there like in Windows Vista.

- Dmitry Vostokov @ DumpAnalysis.org -

In Windows Vista and Server 2008 LPC is implemented using undocumented Advanced Local Procedure Calls (ALPC) and the new formatting of the !process 0 ff output shows the owner of the port: 

THREAD fffffa801afae6f0  Cid 033c.3c40  Teb: 000007fffffa0000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa801afaea80  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8800f4cc950 : queued at port fffffa8017dbb060 : owned by process fffffa801807b8f0

Instead of !lpc we should use !alpc extension command: 

0: kd> !lpc message fffff8800f4cc950
LPC is now emulated over ALPC. Use !alpc
Arguments:

   /?                   - Print help
   /P [/V] PortAddress  - Dump the ALPC port information
   /M [/V] MsgAddress   - Searches the message and dumps the message information
   /R [/V] ResAddress   - Dumps the information regarding resources
   /G [/V]              - Dumps the global information

0: kd> !alpc /M fffff8800f4cc950

Message @ fffff8800f4cc950
  MessageID             : 0x0BFC (3068)
  CallbackID            : 0x1FEF27 (2092839)
  SequenceNumber        : 0x0004D4D4 (316628)
  Type                  : LPC_REQUEST
  DataLength            : 0x0068 (104)
  TotalLength           : 0x0090 (144)
  Canceled              : No
  Release               : No
  ReplyWaitReply        : No
  Continuation          : Yes
  OwnerPort             : fffffa8018a95e60 [ALPC_CLIENT_COMMUNICATION_PORT]
  WaitingThread         : fffffa801afae6f0
  QueueType             : ALPC_MSGQUEUE_PENDING
  QueuePort             : fffffa8017dbb060 [ALPC_CONNECTION_PORT]
  QueuePortOwnerProcess : fffffa801807b8f0 (svchost.exe)
  ServerThread          : fffffa801aeec060
  QuotaCharged          : No
  CancelQueuePort       : 0000000000000000
  CancelSequencePort    : 0000000000000000
  CancelSequenceNumber  : 0×00000000 (0)
  ClientContext         : 00000000025599e0
  ServerContext         : 0000000000000000
  PortContext           : 000000000030d640
  CancelPortContext     : 0000000000000000
  SecurityData          : 0000000000000000
  View                  : 0000000000000000

- Dmitry Vostokov @ DumpAnalysis.org -

Looking at one kernel memory dump from x64 Windows Server 2008 I noticed this API call (shown in blue):

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffffa60`138f4720 fffff800`01875f8a nt!KiSwapContext+0x7f
fffffa60`138f4860 fffff800`0187776a nt!KiSwapThread+0x2fa
fffffa60`138f48d0 fffff800`01ab16d6 nt!KeWaitForSingleObject+0x2da
fffffa60`138f4960 fffff800`01ab1667 nt!FsRtlCancellableWaitForMultipleObjects+0x62
fffffa60`138f49c0 fffffa60`06c515e0 nt!FsRtlCancellableWaitForSingleObject+0x27
fffffa60`138f4a00 fffffa60`06c611dc rdbss!RxWaitForStableCondition+0x11c
fffffa60`138f4a40 fffffa60`06c61c07 rdbss!RxFindOrCreateConnections+0x44c
fffffa60`138f4b20 fffffa60`06c56840 rdbss!RxConstructVirtualNetRoot+0xb7
fffffa60`138f4bc0 fffffa60`06c6381a rdbss!RxFindOrConstructVirtualNetRoot+0x594
fffffa60`138f4d30 fffffa60`06c54c42 rdbss!RxCreateTreeConnect+0x13e
fffffa60`138f4dc0 fffffa60`06c2fbf6 rdbss!RxCommonCreate+0x20a
fffffa60`138f4e80 fffffa60`06c5191a rdbss!RxFsdCommonDispatch+0x786
fffffa60`138f4f70 fffffa60`07e4f21f rdbss!RxFsdDispatch+0x21a
fffffa60`138f4fe0 fffffa60`011e05f5 mrxsmb!MRxSmbFsdDispatch+0xbf
fffffa60`138f5020 fffffa60`011e0130 mup!MupiCallUncProvider+0x159
fffffa60`138f5090 fffffa60`011e17af mup!MupStateMachine+0x120
fffffa60`138f50e0 fffffa60`00d200b4 mup!MupCreate+0x2c3
fffffa60`138f5160 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
[...]
3rd party filter drivers
[...]
fffffa60`138f55a0 fffff800`01aefa59 nt!IopParseDevice+0x5e3
fffffa60`138f5740 fffff800`01af3944 nt!ObpLookupObjectName+0x5eb
fffffa60`138f5850 fffff800`01affee0 nt!ObOpenObjectByName+0x2f4
fffffa60`138f5920 fffff800`01b00a0c nt!IopCreateFile+0x290
fffffa60`138f59c0 fffff800`0186fdf3 nt!NtCreateFile+0x78
fffffa60`138f5a50 fffff800`01870300 nt!KiSystemServiceCopyEnd+0x13
fffffa60`138f5c58 fffffa60`06c91a5e nt!KiServiceLinkage
fffffa60`138f5c60 fffff800`018913d1 dfsc!DfscConnOpenIpcConnectionCallout+0xbe
fffffa60`138f5d20 fffffa60`06c91d08 nt!KeExpandKernelStackAndCalloutEx+0×2e1
fffffa60`138f5db0 fffffa60`06c9bbcc dfsc!DfscGetIpcConnection+0×1f0
fffffa60`138f5e30 fffffa60`06c9bb21 dfsc!DfscRmGetReferral+0×78
fffffa60`138f5ea0 fffffa60`06c91470 dfsc!DfscGetDomainDCReferral+0×31
fffffa60`138f5ef0 fffffa60`06c917ec dfsc!DfscRmValidateDomainIterate+0×5c
fffffa60`138f5f40 fffffa60`06c915f5 dfsc!DfscValidateReferral+0xa0
fffffa60`138f5fb0 fffffa60`06c917ec dfsc!DfscRmValidateRootGetParent+0×75
fffffa60`138f5fe0 fffffa60`06c90825 dfsc!DfscValidateReferral+0xa0
fffffa60`138f6050 fffffa60`06c93905 dfsc!DfscCmValidateState+0×79
fffffa60`138f6090 fffffa60`06c9e759 dfsc!DfscSurrogateCreate+0×7d
fffffa60`138f6100 fffffa60`011e03ab dfsc!DfscSurrogatePreProcess+0xb9
fffffa60`138f6130 fffffa60`011e014f mup!MupCallSurrogatePrePost+0×10b
fffffa60`138f6190 fffffa60`011e17af mup!MupStateMachine+0×13f
fffffa60`138f61e0 fffffa60`00d200b4 mup!MupCreate+0×2c3
fffffa60`138f6260 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
[…]
3rd party filter drivers
[…]
fffffa60`138f6610 fffff800`01aefa59 nt!IopParseDevice+0×5e3
fffffa60`138f67b0 fffff800`01af3944 nt!ObpLookupObjectName+0×5eb
fffffa60`138f68c0 fffff800`01ac22f1 nt!ObOpenObjectByName+0×2f4
fffffa60`138f6990 fffff800`0186fdf3 nt!NtQueryAttributesFile+0×134
fffffa60`138f6c20 00000000`77285e4a nt!KiSystemServiceCopyEnd+0×13

This API is mentioned in the following presentation and document and can also be found in WDK:

PPT: Windows Memory Management Advances

DOC: Advances in Memory Management 

KeExpandKernelStackAndCallout

Its 3rd parameter is the stack size and we can see it used in disassembly where r8d register is used for 3rd parameter according to x64 calling convention and rcx is used for the first parameter, a function procedure to be executed with a guaranteed kernel stack size:

0: kd> kv 100
Child-SP          RetAddr           : Args to Child                                                           : Call Site
[...]
fffffa60`138f5c60 fffff800`018913d1 : 00000000`00000000 fffff880`10d6d3f8 00000000`00000000 00000000`00000000 : dfsc!DfscConnOpenIpcConnectionCallout+0xbe
fffffa60`138f5d20 fffffa60`06c91d08 : fffffa60`06c919a0 fffffa60`138f5df0 fffff880`102128d0 fffffa60`138f5f10 : nt!KeExpandKernelStackAndCalloutEx+0×2e1
fffffa60`138f5db0 fffffa60`06c9bbcc : 00000000`00000000 fffff880`10d6d3f8 00000000`00000000 fffff880`10d6d460 : dfsc!DfscGetIpcConnection+0×1f0
[…]

0: kd> ub fffffa60`06c91d08
dfsc!DfscGetIpcConnection+0×1c6:
fffffa60`06c91cde xor     r9d,r9d
fffffa60`06c91ce1 mov     qword ptr [rsp+50h],rax
fffffa60`06c91ce6 mov     rax,qword ptr [dfsc!DfscGlobalData+0×138 (fffffa60`06c8d758)]
fffffa60`06c91ced mov     r8d,6000h
fffffa60`06c91cf3 mov     qword ptr [rsp+40h],rdi
fffffa60`06c91cf8 mov     byte ptr [rsp+58h],r11b
fffffa60`06c91cfd mov     qword ptr [rsp+20h],rax
fffffa60`06c91d02 call    qword ptr [dfsc!_imp_KeExpandKernelStackAndCalloutEx (fffffa60`06c8b0d0)]

0: kd> ub fffffa60`06c91cde
dfsc!DfscGetIpcConnection+0x199:
fffffa60`06c91cb1 488b88b8000000  mov     rcx,qword ptr [rax+0B8h]
fffffa60`06c91cb8 0fba61100a      bt      dword ptr [rcx+10h],0Ah
fffffa60`06c91cbd 450f42df        cmovb   r11d,r15d
fffffa60`06c91cc1 488b4338        mov     rax,qword ptr [rbx+38h]
fffffa60`06c91cc5 488d542440      lea     rdx,[rsp+40h]
fffffa60`06c91cca 488d0dcffcffff  lea     rcx,[dfsc!DfscConnOpenIpcConnectionCallout (fffffa60`06c919a0)]
fffffa60`06c91cd1 4889442448      mov     qword ptr [rsp+48h],rax
fffffa60`06c91cd6 488d842490000000 lea     rax,[rsp+90h]

It is good sign to see it used in file system stacks because in the past the fixed kernel stacks resulted in stack overflows and double faults:

Stack Overflow Pattern (kernel mode)

- Dmitry Vostokov @ DumpAnalysis.org -

I’m sorry to announce that the book has been delayed and the publication date has been changed to 30th of November, 2009. I promise this delay is the last one and kindly ask you to be patient. As a bonus or compensation for it, the book will also cover Windows 7.

- Dmitry Vostokov @ DumpAnalysis.org -

Just found on Amazon this forthcoming book:

Windows® Sysinternals Administrator’s Reference (Inside Out)

- Dmitry Vostokov @ DumpAnalysis.org -

One of the good outcomes of the previously announced restructuring: the book Crash Dump Analysis for System Administrators (Windows edition) has been prioritized to be published on 30th of November, 2009 due to the overwhelming demand. The book will soon be available for pre-orders.

- Dmitry Vostokov @ DumpAnalysis.org -