Crash Dump Analysis Patterns (Part 27)
Friday, September 14th, 2007Sometimes a problem can be identified not from a single Stack Trace pattern but from a Stack Trace Collection.
These include Coupled Processes, Procedure Call Chains and Blocked Threads. All of them will be discussed in subsequent parts and in this part I only discuss various methods to list stack traces.
- Process dumps including various process minidumps:
~*kv command lists all process threads
!findstack module[!symbol] 2 command filters all stack traces to show ones containing module or module!symbol
- Kernel minidumps:
have only one problem thread. kv command or its variant is suffice.
-
Kernel and complete memory dumps:
!process 0 ff command lists all processes and their threads including user space process thread stacks for complete memory dumps. This command is valid for Windows XP and later. For older systems use WinDbg scripts
!stacks 2 [module[!symbol]] command shows kernel mode stack traces and you can filter the output based on module or module!symbol. Filtering is valid only for dumps from Windows XP and later systems
~[ProcessorN]s;.reload /user;kv command sequence shows stack trace for the running thread on the specified processor.
!sprocess 0n<SessionId> ff lists all processes and their threads for the specified [terminal services] session.
!for_each_thread “.thread /r /p @#Thread; kv” allows execution of stack trace command variants per each thread. The following script takes the advantage of this command to list complete stack traces from an x64 system.
The processor change command is illustrated in this example:
0: kd> ~2s
2: kd> k
ChildEBP RetAddr
eb42bd58 00000000 nt!KiIdleLoop+0x14
2: kd> ~1s;.reload /user;k
Loading User Symbols
...
ChildEBP RetAddr
be4f8c30 eb091f43 i8042prt!I8xProcessCrashDump+0x53
be4f8c8c 8046bfe2 i8042prt!I8042KeyboardInterruptService+0x15d
be4f8c8c 8049470f nt!KiInterruptDispatch+0x32
be4f8d54 80468389 nt!NtSetEvent+0x71
be4f8d54 77f8290a nt!KiSystemService+0xc9
081cfefc 77f88266 ntdll!ZwSetEvent+0xb
081cff0c 77f881b1 ntdll!RtlpUnWaitCriticalSection+0x1b
081cff14 1b00c7d1 ntdll!RtlLeaveCriticalSection+0x1d
081cff4c 1b0034da msjet40!Database::ReadPages+0x81
081cffb4 7c57b3bc msjet40!System::WorkerThread+0x115
081cffec 00000000 KERNEL32!BaseThreadStart+0x52
Example of !findstack command (process dump):
0:000> !findstack kernel32!RaiseException 2
Thread 000, 1 frame(s) match
* 00 0013b3f8 72e8d3ef kernel32!RaiseException+0x53
01 0013b418 72e9a26b msxml3!Exception::raiseException+0x5f
02 0013b424 72e8ff00 msxml3!Exception::_throwError+0x22
03 0013b46c 72e6abaa msxml3!COMSafeControlRoot::getBaseURL+0x3d
04 0013b4bc 72e6a888 msxml3!Document::loadXML+0x82
05 0013b510 64b73a9b msxml3!DOMDocumentWrapper::loadXML+0x5a
06 0013b538 64b74eb6 iepeers!CPersistUserData::initXMLCache+0xa6
07 0013b560 77d0516e iepeers!CPersistUserData::load+0xfc
08 0013b57c 77d14abf oleaut32!DispCallFunc+0x16a
...
...
...
66 0013fec8 0040243d shdocvw!IEWinMain+0x129
67 0013ff1c 00402744 iexplore!WinMain+0x316
68 0013ffc0 77e6f23b iexplore!WinMainCRTStartup+0x182
69 0013fff0 00000000 kernel32!BaseProcessStart+0x23
Example of !stacks command (kernel dump):
2: kd> !stacks 2 nt!PspExitThread
Proc.Thread .Thread Ticks ThreadState Blocker
[8a390818 System]
[8a1bbbf8 smss.exe]
[8a16cbf8 csrss.exe]
[89c14bf0 winlogon.exe]
[89dda630 services.exe]
[89c23af0 lsass.exe]
[8a227470 svchost.exe]
[89f03bb8 svchost.exe]
[89de3820 svchost.exe]
[89d09b60 svchost.exe]
[89c03530 ccEvtMgr.exe]
[89b8f4f0 ccSetMgr.exe]
[89dfe8c0 SPBBCSvc.exe]
[89c9db18 svchost.exe]
[89dfa268 spoolsv.exe]
[89dfa6b8 msdtc.exe]
[89df38f0 CpSvc.exe]
[89d97d88 DefWatch.exe]
[89e04020 IBMSPSVC.EXE]
[89b54710 IBMSPREM.EXE]
[89d9e4b0 IBMSPREM.EXE]
[89c2c4e8 svchost.exe]
[89d307c0 SavRoam.exe]
[89bfcd88 Rtvscan.exe]
[89b53b60 uphclean.exe]
[89c24020 AgentSVC.exe]
[89d75b60 sAginst.exe]
[89cf0d88 CdfSvc.exe]
[89d87020 cdmsvc.exe]
[89dafd88 ctxxmlss.exe]
[89d8dd88 encsvc.exe]
[89d06d88 ImaSrv.exe]
[89d37b60 mfcom.exe]
[89c8bb18 SmaService.exe]
[89d2ba80 svchost.exe]
[89ce8630 XTE.exe]
[89b64b60 XTE.exe]
[89b7c680 ctxcpusched.exe]
[88d94a88 ctxcpuusync.exe]
[89ba5418 unsecapp.exe]
[89d846e0 wmiprvse.exe]
[89cda9d8 ctxwmisvc.exe]
[88d6cb78 logon.scr]
[88ba0a70 csrss.exe]
[88961968 winlogon.exe]
[8865f740 rdpclip.exe]
[8858db20 wfshell.exe]
[88754020 explorer.exe]
[88846d88 BacsTray.exe]
[886b6180 ccApp.exe]
[884bc020 fppdis3a.exe]
[885cb350 ctfmon.exe]
[888bb918 cscript.exe]
[8880b3c8 cscript.exe]
[88ad2950 csrss.exe]
b68.00215c 88930020 0000000 RUNNING nt!KeBugCheckEx+0x1b
nt!MiCheckSessionPoolAllocations+0xe3
nt!MiDereferenceSessionFinal+0x183
nt!MmCleanProcessAddressSpace+0x6b
nt!PspExitThread+0x5f1
nt!PspTerminateThreadByPointer+0x4b
nt!PspSystemThreadStartup+0x3c
nt!KiThreadStartup+0x16
[88629310 winlogon.exe]
[88a4d9b0 csrss.exe]
[88d9f8b0 winlogon.exe]
[88cd5840 wfshell.exe]
[8a252440 OUTLOOK.EXE]
[8a194bf8 WINWORD.EXE]
[88aabd20 ctfmon.exe]
[889ef440 EXCEL.EXE]
[88bec838 HogiaGUI2.exe]
[88692020 csrss.exe]
[884dd508 winlogon.exe]
[88be1d88 wfshell.exe]
[886a7d88 OUTLOOK.EXE]
[889baa70 WINWORD.EXE]
[8861e3d0 ctfmon.exe]
[887bbb68 EXCEL.EXE]
[884e4020 csrss.exe]
[8889d218 winlogon.exe]
[887c8020 wfshell.exe]
Threads Processed: 1101
What if we have a list of processes from a complete memory dump by using !process 0 0 command and we want to interrogate the specific process? In this case we need to switch to that process and reload user space symbol files (.process /r /p address).
There is also a separate command to reload user space symbol files any time (.reload /user).
After switching we can list threads (!process address), dump or search process virtual memory, etc. For example:
1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 890a3320 SessionId: 0 Cid: 0008 Peb: 00000000 ParentCid: 0000
DirBase: 00030000 ObjectTable: 890a3e08 TableSize: 405.
Image: System
PROCESS 889dfd60 SessionId: 0 Cid: 0144 Peb: 7ffdf000 ParentCid: 0008
DirBase: 0b9e7000 ObjectTable: 889fdb48 TableSize: 212.
Image: SMSS.EXE
PROCESS 890af020 SessionId: 0 Cid: 0160 Peb: 7ffdf000 ParentCid: 0144
DirBase: 0ce36000 ObjectTable: 8898e308 TableSize: 747.
Image: CSRSS.EXE
PROCESS 8893d020 SessionId: 0 Cid: 0178 Peb: 7ffdf000 ParentCid: 0144
DirBase: 0d33b000 ObjectTable: 890ab4c8 TableSize: 364.
Image: WINLOGON.EXE
PROCESS 88936020 SessionId: 0 Cid: 0194 Peb: 7ffdf000 ParentCid: 0178
DirBase: 0d7d5000 ObjectTable: 88980528 TableSize: 872.
Image: SERVICES.EXE
PROCESS 8897f020 SessionId: 0 Cid: 01a0 Peb: 7ffdf000 ParentCid: 0178
DirBase: 0d89d000 ObjectTable: 889367c8 TableSize: 623.
Image: LSASS.EXE
1: kd> .process /r /p 8893d020
Implicit process is now 8893d020
Loading User Symbols
...
1: kd> !process 8893d020
PROCESS 8893d020 SessionId: 0 Cid: 0178 Peb: 7ffdf000 ParentCid: 0144
DirBase: 0d33b000 ObjectTable: 890ab4c8 TableSize: 364.
Image: WINLOGON.EXE
VadRoot 8893a508 Clone 0 Private 1320. Modified 45178. Locked 0.
DeviceMap 89072448
Token e392f8d0
ElapsedTime 9:54:06.0882
UserTime 0:00:00.0071
KernelTime 0:00:00.0382
QuotaPoolUsage[PagedPool] 34828
QuotaPoolUsage[NonPagedPool] 43440
Working Set Sizes (now,min,max) (737, 50, 345) (2948KB, 200KB, 1380KB)
PeakWorkingSetSize 2764
VirtualSize 46 Mb
PeakVirtualSize 52 Mb
PageFaultCount 117462
MemoryPriority FOREGROUND
BasePriority 13
CommitCharge 1861
THREAD 8893dda0 Cid 178.15c Teb: 7ffde000 Win32Thread: a2034908 WAIT: (WrUserRequest) UserMode Non-Alertable
8893bee0 SynchronizationEvent
Not impersonating
Owning Process 8893d020
Wait Start TickCount 29932455 Elapsed Ticks: 7
Context Switch Count 28087 LargeStack
UserTime 0:00:00.0023
KernelTime 0:00:00.0084
Start Address winlogon!WinMainCRTStartup (0x0101cbb0)
Stack Init eb1b0000 Current eb1afcc8 Base eb1b0000 Limit eb1ac000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
eb1afce0 8042d893 00000000 a2034908 00000001 nt!KiSwapThread+0x1b1
eb1afd08 a00019c2 8893bee0 0000000d 00000001 nt!KeWaitForSingleObject+0x1a3
eb1afd44 a0013993 000020ff 00000000 00000001 win32k!xxxSleepThread+0x18a
eb1afd54 a001399f 0006fdd8 80468389 00000000 win32k!xxxWaitMessage+0xe
eb1afd5c 80468389 00000000 00000000 00000000 win32k!NtUserWaitMessage+0xb
eb1afd5c 77e58b53 00000000 00000000 00000000 nt!KiSystemService+0xc9
0006fdd0 77e33630 00000000 00000000 0000ffff USER32!NtUserWaitMessage+0xb
0006fe04 77e44327 000100d2 00000000 00000010 USER32!DialogBox2+0x216
0006fe28 77e38d37 76b90000 76c75c78 00000000 USER32!InternalDialogBox+0xd1
0006fe48 77e39eba 76b90000 76c75c78 00000000 USER32!DialogBoxIndirectParamAorW+0x34
0006fe6c 01011749 76b90000 00000578 00000000 USER32!DialogBoxParamW+0x3d
0006fea8 01018bd3 000755e8 76b90000 00000578 winlogon!TimeoutDialogBoxParam+0x27
0006fee0 76b93701 000755e8 76b90000 00000578 winlogon!WlxDialogBoxParam+0×7b
0006ff08 010164c6 0008d0e0 5ffa0000 000755e8 3rdPartyGINA!WlxDisplaySASNotice+0×43
0006ff20 01014960 000755e8 00000005 00072c9c winlogon!MainLoop+0×96
0006ff58 0101cd06 00071fc8 00000000 00072c9c winlogon!WinMain+0×37a
0006fff4 00000000 7ffdf000 000000c8 00000100 winlogon!WinMainCRTStartup+0×156
THREAD 88980020 Cid 178.188 Teb: 7ffdc000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Alertable
88980108 NotificationTimer
Not impersonating
Owning Process 8893d020
Wait Start TickCount 29930810 Elapsed Ticks: 1652
Context Switch Count 15638
UserTime 0:00:00.0000
KernelTime 0:00:00.0000
Start Address KERNEL32!BaseThreadStartThunk (0x7c57b740)
Win32 Start Address ntdll!RtlpTimerThread (0x77faa02d)
Stack Init bf6f7000 Current bf6f6cc4 Base bf6f7000 Limit bf6f4000 Call 0
Priority 13 BasePriority 13 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
bf6f6cdc 8042d340 bf6f6d64 00bfffac 00bfffac nt!KiSwapThread+0x1b1
bf6f6d04 8052aac9 8046c101 00000001 bf6f6d34 nt!KeDelayExecutionThread+0x182
bf6f6d54 80468389 00000001 00bfffac 00000000 nt!NtDelayExecution+0x7f
bf6f6d54 77f82831 00000001 00bfffac 00000000 nt!KiSystemService+0xc9
00bfff9c 77f842c4 00000001 00bfffac 00000000 ntdll!NtDelayExecution+0xb
00bfffb4 7c57b3bc 0006fe60 00000000 00000000 ntdll!RtlpTimerThread+0x42
00bfffec 00000000 77faa02d 0006fe60 00000000 KERNEL32!BaseThreadStart+0x52
1: kd> dds 0006fee0
0006fee0 0006ff08
0006fee4 76b93701 3rdPartyGINA!WlxDisplaySASNotice+0x43
0006fee8 000755e8
0006feec 76b90000 3rdParty
0006fef0 00000578
0006fef4 00000000
0006fef8 76b9370b 3rdParty!WlxDisplaySASNotice+0x4d
0006fefc 0008d0e0
0006ff00 00000008
0006ff04 00000080
0006ff08 0006ff20
0006ff0c 010164c6 winlogon!MainLoop+0x96
0006ff10 0008d0e0
0006ff14 5ffa0000
0006ff18 000755e8
0006ff1c 00000000
0006ff20 0006ff58
0006ff24 01014960 winlogon!WinMain+0x37a
0006ff28 000755e8
0006ff2c 00000005
0006ff30 00072c9c
0006ff34 00000001
0006ff38 000001bc
0006ff3c 00000005
0006ff40 00000001
0006ff44 0000000d
0006ff48 00000000
0006ff4c 00000000
0006ff50 00000000
0006ff54 0000ffe4
0006ff58 0006fff4
0006ff5c 0101cd06 winlogon!WinMainCRTStartup+0x156
- Dmitry Vostokov @ DumpAnalysis.org -