32-bit stack traces from x64 complete dumps
In the past I was not able or didn’t know how to view 32-bit process thread stacks when looking at a complete memory dump from x64 Windows. So I had to request user dumps. Now I want to share a technique a reader of my blog (Yuhong Bao) suggested: to use .thread WinDbg command with /w option. Here are additional steps that I found necessary when playing with my test complete memory dump from x64 Windows Server 2003 SP2 (I used the latest version of WinDbg from 64-bit Debugging Tools for Windows):
0. Find a 32-bit process of interest:
kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffadfe7afd8e0
SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 0014a000 ObjectTable: fffffa8000000c10 HandleCount: 736.
Image: System
PROCESS fffffadfe6b14040
SessionId: none Cid: 0130 Peb: 7fffffd8000 ParentCid: 0004
DirBase: 353c0000 ObjectTable: fffffa80009104a0 HandleCount: 19.
Image: smss.exe
PROCESS fffffadfe65cec20
SessionId: 0 Cid: 0160 Peb: 7fffffde000 ParentCid: 0130
DirBase: 30210000 ObjectTable: fffffa80006a4d80 HandleCount: 732.
Image: csrss.exe
PROCESS fffffadfe73b7040
SessionId: 0 Cid: 0270 Peb: 7fffffdc000 ParentCid: 0130
DirBase: 302b6000 ObjectTable: fffffa8000520710 HandleCount: 751.
Image: winlogon.exe
PROCESS fffffadfe737d040
SessionId: 0 Cid: 02a0 Peb: 7fffffd7000 ParentCid: 0270
DirBase: 0060d000 ObjectTable: fffffa80008df6a0 HandleCount: 339.
Image: services.exe
PROCESS fffffadfe6574040
SessionId: 0 Cid: 02ac Peb: 7fffffd5000 ParentCid: 0270
DirBase: 0070d000 ObjectTable: fffffa80008e16a0 HandleCount: 510.
Image: lsass.exe
PROCESS fffffadfe7860040
SessionId: 0 Cid: 0364 Peb: 7fffffd7000 ParentCid: 02a0
DirBase: 0935e000 ObjectTable: fffffa8000969710 HandleCount: 87.
Image: svchost.exe
[...]
PROCESS fffffadfe751d040
SessionId: 0 Cid: 0bcc Peb: 7efdf000 ParentCid: 0abc
DirBase: 18861000 ObjectTable: fffffa8001ecbc30 HandleCount: 326.
Image: Application32.exe
[...]
1. Switch to the process context:
kd> .process /r /p fffffadfe751d040
Implicit process is now fffffadf`e751d040
Loading User Symbols
Stacks traces are 64-bit:
kd> !process fffffadfe751d040
PROCESS fffffadfe751d040
SessionId: 0 Cid: 0bcc Peb: 7efdf000 ParentCid: 0abc
DirBase: 18861000 ObjectTable: fffffa8001ecbc30 HandleCount: 326.
Image: Application32.exe
VadRoot fffffadfe7550ae0 Vads 160 Clone 0 Private 1616. Modified 1675. Locked 0.
DeviceMap fffffa800210e600
Token fffffa80028ef060
ElapsedTime 21:57:59.125
UserTime 00:00:00.718
KernelTime 00:00:00.953
QuotaPoolUsage[PagedPool] 185704
QuotaPoolUsage[NonPagedPool] 20080
Working Set Sizes (now,min,max) (3021, 50, 345) (12084KB, 200KB, 1380KB)
PeakWorkingSetSize 3696
VirtualSize 93 Mb
PeakVirtualSize 104 Mb
PageFaultCount 12097
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 2051
THREAD fffffadfe664e040 Cid 0bcc.0bdc Teb: 000000007efdb000 Win32Thread: fffff97ff4898bd0 WAIT: (Unknown) UserMode Non-Alertable
fffffadfe73bac40 SynchronizationEvent
fffffadfe6b69790 SynchronizationEvent
Not impersonating
DeviceMap fffffa800210e600
Owning Process fffffadfe751d040 Image: Application32.exe
Attached Process N/A Image: N/A
Wait Start TickCount 4153935 Ticks: 912354 (0:03:57:35.531)
Context Switch Count 8088 LargeStack
UserTime 00:00:00.343
KernelTime 00:00:00.593
Win32 Start Address Application32 (0x00000000004077ec)
Start Address 0x0000000077d59620
Stack Init fffffadfdede7e00 Current fffffadfdede7250
Base fffffadfdede8000 Limit fffffadfdede2000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
Child-SP RetAddr Call Site
fffffadf`dede7290 fffff800`0103b0a3 nt!KiSwapContext+0x85
fffffadf`dede7410 fffff800`0103af8a nt!KiSwapThread+0xc3
fffffadf`dede7450 fffff800`012b9958 nt!KeWaitForMultipleObjects+0x5ec
fffffadf`dede74f0 fffff800`012e63ec nt!ObpWaitForMultipleObjects+0x325
fffffadf`dede79b0 fffff800`0104113d nt!NtWaitForMultipleObjects32+0xcc
fffffadf`dede7c00 00000000`78b83d44 nt!KiSystemServiceCopyEnd+0x3 (TrapFrame @ fffffadf`dede7c70)
00000000`0012edc8 00000000`6b006a5a wow64cpu!WaitForMultipleObjects32+0x3a
00000000`0012ee70 00000000`6b005e0d wow64!RunCpuSimulation+0xa
00000000`0012eea0 00000000`77ed8030 wow64!Wow64LdrpInitialize+0x2ed
00000000`0012f6d0 00000000`77ed582f ntdll!LdrpInitializeProcess+0x1538
00000000`0012f9d0 00000000`77ef30a5 ntdll!LdrpInitialize+0x18f
00000000`0012fab0 00000000`77d59620 ntdll!KiUserApcDispatcher+0x15 (TrapFrame @ 00000000`0012fe18)
[...]
2. Load WOW64 extension
kd> .load wow64exts
3. Set the current thread and switch to x86 context:
kd> .thread /w fffffadfe664e040
Implicit thread is now fffffadf`e664e040
x86 context set
4. Sometimes reloading symbols is necessary:
kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols
5. Now we can get our stack trace (it is a bit rough because Application32.exe symbols were not available)
kd:x86> kv 100
ChildEBP RetAddr Args to Child
002cfd94 7d4e286c 00000002 002cfde0 00000001 ntdll_7d600000!NtWaitForMultipleObjects+0x15 (FPO: [5,0,0])
002cfe3c 7d94d299 00000002 002cfe64 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
002cfe98 7d94d327 00000001 002d8148 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x152 (FPO: [5,13,0])
*** ERROR: Module load completed but symbols could not be loaded for Application32.exe
002cfeb4 00408081 00000001 002d8148 00000000 USER32!MsgWaitForMultipleObjects+0x1f (FPO: [5,0,0])
WARNING: Stack unwind information not available. Following frames may be wrong.
002cff00 00407d4b ffffffff 00408b78 004010ee Application32+0x8081
002cff08 00408b78 004010ee 004352e0 004352e0 Application32+0x7d4b
002cff0c 004010ee 004352e0 004352e0 0042f004 Application32+0x8b78
00408b78 90909090 90c3c033 90909090 90909090 Application32+0x10ee
00408b7c 90c3c033 90909090 90909090 90909090 0x90909090
00408b80 90909090 90909090 90909090 433aa0a1 0x90c3c033
[...]
6. We can also access raw stack trace if we need to see 32-bit execution residue and reconstruct partial stack traces:
kd:x86> !teb
Wow64 TEB32 at 000000007efdd000
[...]
Wow64 TEB at 000000007efdb000
ExceptionList: 000000007efdd000
StackBase: 0000000000130000
StackLimit: 000000000012a000
SubSystemTib: 0000000000000000
FiberData: 0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self: 000000007efdb000
EnvironmentPointer: 0000000000000000
ClientId: 0000000000000bcc . 0000000000000bdc
RpcHandle: 0000000000000000
Tls Storage: 0000000000000000
PEB Address: 000000007efdf000
LastErrorValue: 6
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
kd:x86> dds 000000000012a000 0000000000130000
[...]
- Dmitry Vostokov @ DumpAnalysis.org -
February 9th, 2010 at 10:35 pm
[…] I wrote on how to get a 32-bit stack trace from a 32-bit process thread on an x64 system. There are situations when we are interested in all such stack traces, for example, from a complete […]