Crash Dump Analysis

Sponsored link: Memory Dump Analysis Services

Debugging Experts Magazine Online

In the past I was not able or didn’t know how to view 32-bit process thread stacks when looking at a complete memory dump from x64 Windows. So I had to request user dumps. Now I want to share a technique a reader of my blog (Yuhong Bao) suggested: to use .thread WinDbg command with /w option. Here are additional steps that I found necessary when playing with my test complete memory dump from x64 Windows Server 2003 SP2 (I used the latest version of WinDbg from 64-bit Debugging Tools for Windows):

0. Find a 32-bit process of interest:

kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffadfe7afd8e0
    SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 0014a000  ObjectTable: fffffa8000000c10  HandleCount: 736.
    Image: System

PROCESS fffffadfe6b14040
    SessionId: none  Cid: 0130    Peb: 7fffffd8000  ParentCid: 0004
    DirBase: 353c0000  ObjectTable: fffffa80009104a0  HandleCount:  19.
    Image: smss.exe

PROCESS fffffadfe65cec20
    SessionId: 0  Cid: 0160    Peb: 7fffffde000  ParentCid: 0130
    DirBase: 30210000  ObjectTable: fffffa80006a4d80  HandleCount: 732.
    Image: csrss.exe

PROCESS fffffadfe73b7040
    SessionId: 0  Cid: 0270    Peb: 7fffffdc000  ParentCid: 0130
    DirBase: 302b6000  ObjectTable: fffffa8000520710  HandleCount: 751.
    Image: winlogon.exe

PROCESS fffffadfe737d040
    SessionId: 0  Cid: 02a0    Peb: 7fffffd7000  ParentCid: 0270
    DirBase: 0060d000  ObjectTable: fffffa80008df6a0  HandleCount: 339.
    Image: services.exe

PROCESS fffffadfe6574040
    SessionId: 0  Cid: 02ac    Peb: 7fffffd5000  ParentCid: 0270
    DirBase: 0070d000  ObjectTable: fffffa80008e16a0  HandleCount: 510.
    Image: lsass.exe

PROCESS fffffadfe7860040
    SessionId: 0  Cid: 0364    Peb: 7fffffd7000  ParentCid: 02a0
    DirBase: 0935e000  ObjectTable: fffffa8000969710  HandleCount:  87.
    Image: svchost.exe

[...]

PROCESS fffffadfe751d040
    SessionId: 0  Cid: 0bcc    Peb: 7efdf000  ParentCid: 0abc
    DirBase: 18861000  ObjectTable: fffffa8001ecbc30  HandleCount: 326.
    Image: Application32.exe

[...]

1. Switch to the process context:

kd> .process /r /p fffffadfe751d040
Implicit process is now fffffadf`e751d040
Loading User Symbols

Stacks traces are 64-bit:

kd> !process fffffadfe751d040
PROCESS fffffadfe751d040
    SessionId: 0  Cid: 0bcc    Peb: 7efdf000  ParentCid: 0abc
    DirBase: 18861000  ObjectTable: fffffa8001ecbc30  HandleCount: 326.
    Image: Application32.exe
    VadRoot fffffadfe7550ae0 Vads 160 Clone 0 Private 1616. Modified 1675. Locked 0.
    DeviceMap fffffa800210e600
    Token                             fffffa80028ef060
    ElapsedTime                       21:57:59.125
    UserTime                          00:00:00.718
    KernelTime                        00:00:00.953
    QuotaPoolUsage[PagedPool]         185704
    QuotaPoolUsage[NonPagedPool]      20080
    Working Set Sizes (now,min,max)  (3021, 50, 345) (12084KB, 200KB, 1380KB)
    PeakWorkingSetSize                3696
    VirtualSize                       93 Mb
    PeakVirtualSize                   104 Mb
    PageFaultCount                    12097
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      2051

THREAD fffffadfe664e040  Cid 0bcc.0bdc  Teb: 000000007efdb000 Win32Thread: fffff97ff4898bd0 WAIT: (Unknown) UserMode Non-Alertable
    fffffadfe73bac40  SynchronizationEvent
    fffffadfe6b69790  SynchronizationEvent
Not impersonating
DeviceMap                 fffffa800210e600
Owning Process            fffffadfe751d040       Image:         Application32.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4153935        Ticks: 912354 (0:03:57:35.531)
Context Switch Count      8088                 LargeStack
UserTime                  00:00:00.343
KernelTime                00:00:00.593
Win32 Start Address Application32 (0x00000000004077ec)
Start Address 0x0000000077d59620
Stack Init fffffadfdede7e00 Current fffffadfdede7250
Base fffffadfdede8000 Limit fffffadfdede2000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffffadf`dede7290 fffff800`0103b0a3 nt!KiSwapContext+0x85
fffffadf`dede7410 fffff800`0103af8a nt!KiSwapThread+0xc3
fffffadf`dede7450 fffff800`012b9958 nt!KeWaitForMultipleObjects+0x5ec
fffffadf`dede74f0 fffff800`012e63ec nt!ObpWaitForMultipleObjects+0x325
fffffadf`dede79b0 fffff800`0104113d nt!NtWaitForMultipleObjects32+0xcc
fffffadf`dede7c00 00000000`78b83d44 nt!KiSystemServiceCopyEnd+0x3 (TrapFrame @ fffffadf`dede7c70)
00000000`0012edc8 00000000`6b006a5a wow64cpu!WaitForMultipleObjects32+0x3a
00000000`0012ee70 00000000`6b005e0d wow64!RunCpuSimulation+0xa
00000000`0012eea0 00000000`77ed8030 wow64!Wow64LdrpInitialize+0x2ed
00000000`0012f6d0 00000000`77ed582f ntdll!LdrpInitializeProcess+0x1538
00000000`0012f9d0 00000000`77ef30a5 ntdll!LdrpInitialize+0x18f
00000000`0012fab0 00000000`77d59620 ntdll!KiUserApcDispatcher+0x15 (TrapFrame @ 00000000`0012fe18)
[...]

2. Load WOW64 extension

kd> .load wow64exts

3. Set the current thread and switch to x86 context:

kd> .thread /w fffffadfe664e040
Implicit thread is now fffffadf`e664e040
x86 context set

4. Sometimes reloading symbols is necessary:

kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols

5. Now we can get our stack trace (it is a bit rough because Application32.exe symbols were not available)

kd:x86> kv 100
ChildEBP          RetAddr           Args to Child                                        
002cfd94 7d4e286c 00000002 002cfde0 00000001 ntdll_7d600000!NtWaitForMultipleObjects+0x15 (FPO: [5,0,0])
002cfe3c 7d94d299 00000002 002cfe64 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
002cfe98 7d94d327 00000001 002d8148 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x152 (FPO: [5,13,0])
*** ERROR: Module load completed but symbols could not be loaded for Application32.exe
002cfeb4 00408081 00000001 002d8148 00000000 USER32!MsgWaitForMultipleObjects+0x1f (FPO: [5,0,0])
WARNING: Stack unwind information not available. Following frames may be wrong.
002cff00 00407d4b ffffffff 00408b78 004010ee Application32+0x8081
002cff08 00408b78 004010ee 004352e0 004352e0 Application32+0x7d4b
002cff0c 004010ee 004352e0 004352e0 0042f004 Application32+0x8b78
00408b78 90909090 90c3c033 90909090 90909090 Application32+0x10ee
00408b7c 90c3c033 90909090 90909090 90909090 0x90909090
00408b80 90909090 90909090 90909090 433aa0a1 0x90c3c033
[...]

6. We can also access raw stack trace if we need to see 32-bit execution residue and reconstruct partial stack traces:  

kd:x86> !teb
Wow64 TEB32 at 000000007efdd000
[...]
Wow64 TEB at 000000007efdb000
    ExceptionList:        000000007efdd000
    StackBase:            0000000000130000
    StackLimit:           000000000012a000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efdb000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000bcc . 0000000000000bdc
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       6
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

kd:x86> dds 000000000012a000 0000000000130000
[...]

- Dmitry Vostokov @ DumpAnalysis.org -

           

Museum of Debugging and Memory Dumps

7/7/2010 - 8/8/2010 Annual Competition: Tell Your Windows Debugging Story

Crash and Hang Analysis Audit Service

CARE: Crash Analysis Report Environment

Crash Dump and Software Trace Analysis Training and Seminars

Access OpenTask Titles on Safari Books Online

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

International Memory Analysts and Debuggers Day:
07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA

Announcements

Coming Soon:

Management Bits: An Anthology from Reductionist Manager

Crash Dump Analysis: Practical Foundations (Windows Edition, Systematic Software Fault Analysis Series)

Debugging Notebook: Essential Concepts, WinDbg Commands and Tools

Crash Dump Analysis for System Administrators and Support Engineers

New Magazines:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers

New Books:

Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3

Memory Dump Analysis Anthology, Volume 3

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

x64 Windows Debugging: Practical Foundations

Also available:

Windows Debugging: Practical Foundations

DLL List Landscape: The Art from Computer Memory Space

Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

WinDbg: A Reference Poster and Learning Cards

Memory Dump Analysis Anthology, Volume 2

Memory Dump Analysis Anthology, Volume 1

New Children's Book:

Baby Turing

This entry was posted on Friday, February 6th, 2009 at 4:30 pm and is filed under Crash Dump Analysis, Debugging, WinDbg Tips and Tricks. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.