Crash Dump Analysis Patterns (Part 27)
Sometimes a problem can be identified not from a single Stack Trace pattern but from a Stack Trace Collection.
These include Coupled Processes, Procedure Call Chains and Blocked Threads. All of them will be discussed in subsequent parts and in this part I only discuss various methods to list stack traces.
- Process dumps including various process minidumps:
~*kv command lists all process threads
!findstack module[!symbol] 2 command filters all stack traces to show ones containing module or module!symbol
- Kernel minidumps:
have only one problem thread. kv command or its variant is suffice.
-
Kernel and complete memory dumps:
!process 0 ff command lists all processes and their threads including user space process thread stacks for complete memory dumps. This command is valid for Windows XP and later. For older systems use WinDbg scripts
!stacks 2 [module[!symbol]] command shows kernel mode stack traces and you can filter the output based on module or module!symbol. Filtering is valid only for dumps from Windows XP and later systems
~[ProcessorN]s;.reload /user;kv command sequence shows stack trace for the running thread on the specified processor.
!sprocess 0n<SessionId> ff lists all processes and their threads for the specified [terminal services] session.
!for_each_thread “.thread /r /p @#Thread; kv” allows execution of stack trace command variants per each thread. The following script takes the advantage of this command to list complete stack traces from an x64 system.
The processor change command is illustrated in this example:
0: kd> ~2s
2: kd> k
ChildEBP RetAddr
eb42bd58 00000000 nt!KiIdleLoop+0x14
2: kd> ~1s;.reload /user;k
Loading User Symbols
...
ChildEBP RetAddr
be4f8c30 eb091f43 i8042prt!I8xProcessCrashDump+0x53
be4f8c8c 8046bfe2 i8042prt!I8042KeyboardInterruptService+0x15d
be4f8c8c 8049470f nt!KiInterruptDispatch+0x32
be4f8d54 80468389 nt!NtSetEvent+0x71
be4f8d54 77f8290a nt!KiSystemService+0xc9
081cfefc 77f88266 ntdll!ZwSetEvent+0xb
081cff0c 77f881b1 ntdll!RtlpUnWaitCriticalSection+0x1b
081cff14 1b00c7d1 ntdll!RtlLeaveCriticalSection+0x1d
081cff4c 1b0034da msjet40!Database::ReadPages+0x81
081cffb4 7c57b3bc msjet40!System::WorkerThread+0x115
081cffec 00000000 KERNEL32!BaseThreadStart+0x52
Example of !findstack command (process dump):
0:000> !findstack kernel32!RaiseException 2
Thread 000, 1 frame(s) match
* 00 0013b3f8 72e8d3ef kernel32!RaiseException+0x53
01 0013b418 72e9a26b msxml3!Exception::raiseException+0x5f
02 0013b424 72e8ff00 msxml3!Exception::_throwError+0x22
03 0013b46c 72e6abaa msxml3!COMSafeControlRoot::getBaseURL+0x3d
04 0013b4bc 72e6a888 msxml3!Document::loadXML+0x82
05 0013b510 64b73a9b msxml3!DOMDocumentWrapper::loadXML+0x5a
06 0013b538 64b74eb6 iepeers!CPersistUserData::initXMLCache+0xa6
07 0013b560 77d0516e iepeers!CPersistUserData::load+0xfc
08 0013b57c 77d14abf oleaut32!DispCallFunc+0x16a
...
...
...
66 0013fec8 0040243d shdocvw!IEWinMain+0x129
67 0013ff1c 00402744 iexplore!WinMain+0x316
68 0013ffc0 77e6f23b iexplore!WinMainCRTStartup+0x182
69 0013fff0 00000000 kernel32!BaseProcessStart+0x23
Example of !stacks command (kernel dump):
2: kd> !stacks 2 nt!PspExitThread
Proc.Thread .Thread Ticks ThreadState Blocker
[8a390818 System]
[8a1bbbf8 smss.exe]
[8a16cbf8 csrss.exe]
[89c14bf0 winlogon.exe]
[89dda630 services.exe]
[89c23af0 lsass.exe]
[8a227470 svchost.exe]
[89f03bb8 svchost.exe]
[89de3820 svchost.exe]
[89d09b60 svchost.exe]
[89c03530 ccEvtMgr.exe]
[89b8f4f0 ccSetMgr.exe]
[89dfe8c0 SPBBCSvc.exe]
[89c9db18 svchost.exe]
[89dfa268 spoolsv.exe]
[89dfa6b8 msdtc.exe]
[89df38f0 CpSvc.exe]
[89d97d88 DefWatch.exe]
[89e04020 IBMSPSVC.EXE]
[89b54710 IBMSPREM.EXE]
[89d9e4b0 IBMSPREM.EXE]
[89c2c4e8 svchost.exe]
[89d307c0 SavRoam.exe]
[89bfcd88 Rtvscan.exe]
[89b53b60 uphclean.exe]
[89c24020 AgentSVC.exe]
[89d75b60 sAginst.exe]
[89cf0d88 CdfSvc.exe]
[89d87020 cdmsvc.exe]
[89dafd88 ctxxmlss.exe]
[89d8dd88 encsvc.exe]
[89d06d88 ImaSrv.exe]
[89d37b60 mfcom.exe]
[89c8bb18 SmaService.exe]
[89d2ba80 svchost.exe]
[89ce8630 XTE.exe]
[89b64b60 XTE.exe]
[89b7c680 ctxcpusched.exe]
[88d94a88 ctxcpuusync.exe]
[89ba5418 unsecapp.exe]
[89d846e0 wmiprvse.exe]
[89cda9d8 ctxwmisvc.exe]
[88d6cb78 logon.scr]
[88ba0a70 csrss.exe]
[88961968 winlogon.exe]
[8865f740 rdpclip.exe]
[8858db20 wfshell.exe]
[88754020 explorer.exe]
[88846d88 BacsTray.exe]
[886b6180 ccApp.exe]
[884bc020 fppdis3a.exe]
[885cb350 ctfmon.exe]
[888bb918 cscript.exe]
[8880b3c8 cscript.exe]
[88ad2950 csrss.exe]
b68.00215c 88930020 0000000 RUNNING nt!KeBugCheckEx+0x1b
nt!MiCheckSessionPoolAllocations+0xe3
nt!MiDereferenceSessionFinal+0x183
nt!MmCleanProcessAddressSpace+0x6b
nt!PspExitThread+0x5f1
nt!PspTerminateThreadByPointer+0x4b
nt!PspSystemThreadStartup+0x3c
nt!KiThreadStartup+0x16
[88629310 winlogon.exe]
[88a4d9b0 csrss.exe]
[88d9f8b0 winlogon.exe]
[88cd5840 wfshell.exe]
[8a252440 OUTLOOK.EXE]
[8a194bf8 WINWORD.EXE]
[88aabd20 ctfmon.exe]
[889ef440 EXCEL.EXE]
[88bec838 HogiaGUI2.exe]
[88692020 csrss.exe]
[884dd508 winlogon.exe]
[88be1d88 wfshell.exe]
[886a7d88 OUTLOOK.EXE]
[889baa70 WINWORD.EXE]
[8861e3d0 ctfmon.exe]
[887bbb68 EXCEL.EXE]
[884e4020 csrss.exe]
[8889d218 winlogon.exe]
[887c8020 wfshell.exe]
Threads Processed: 1101
What if we have a list of processes from a complete memory dump by using !process 0 0 command and we want to interrogate the specific process? In this case we need to switch to that process and reload user space symbol files (.process /r /p address).
There is also a separate command to reload user space symbol files any time (.reload /user).
After switching we can list threads (!process address), dump or search process virtual memory, etc. For example:
1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 890a3320 SessionId: 0 Cid: 0008 Peb: 00000000 ParentCid: 0000
DirBase: 00030000 ObjectTable: 890a3e08 TableSize: 405.
Image: System
PROCESS 889dfd60 SessionId: 0 Cid: 0144 Peb: 7ffdf000 ParentCid: 0008
DirBase: 0b9e7000 ObjectTable: 889fdb48 TableSize: 212.
Image: SMSS.EXE
PROCESS 890af020 SessionId: 0 Cid: 0160 Peb: 7ffdf000 ParentCid: 0144
DirBase: 0ce36000 ObjectTable: 8898e308 TableSize: 747.
Image: CSRSS.EXE
PROCESS 8893d020 SessionId: 0 Cid: 0178 Peb: 7ffdf000 ParentCid: 0144
DirBase: 0d33b000 ObjectTable: 890ab4c8 TableSize: 364.
Image: WINLOGON.EXE
PROCESS 88936020 SessionId: 0 Cid: 0194 Peb: 7ffdf000 ParentCid: 0178
DirBase: 0d7d5000 ObjectTable: 88980528 TableSize: 872.
Image: SERVICES.EXE
PROCESS 8897f020 SessionId: 0 Cid: 01a0 Peb: 7ffdf000 ParentCid: 0178
DirBase: 0d89d000 ObjectTable: 889367c8 TableSize: 623.
Image: LSASS.EXE
1: kd> .process /r /p 8893d020
Implicit process is now 8893d020
Loading User Symbols
...
1: kd> !process 8893d020
PROCESS 8893d020 SessionId: 0 Cid: 0178 Peb: 7ffdf000 ParentCid: 0144
DirBase: 0d33b000 ObjectTable: 890ab4c8 TableSize: 364.
Image: WINLOGON.EXE
VadRoot 8893a508 Clone 0 Private 1320. Modified 45178. Locked 0.
DeviceMap 89072448
Token e392f8d0
ElapsedTime 9:54:06.0882
UserTime 0:00:00.0071
KernelTime 0:00:00.0382
QuotaPoolUsage[PagedPool] 34828
QuotaPoolUsage[NonPagedPool] 43440
Working Set Sizes (now,min,max) (737, 50, 345) (2948KB, 200KB, 1380KB)
PeakWorkingSetSize 2764
VirtualSize 46 Mb
PeakVirtualSize 52 Mb
PageFaultCount 117462
MemoryPriority FOREGROUND
BasePriority 13
CommitCharge 1861
THREAD 8893dda0 Cid 178.15c Teb: 7ffde000 Win32Thread: a2034908 WAIT: (WrUserRequest) UserMode Non-Alertable
8893bee0 SynchronizationEvent
Not impersonating
Owning Process 8893d020
Wait Start TickCount 29932455 Elapsed Ticks: 7
Context Switch Count 28087 LargeStack
UserTime 0:00:00.0023
KernelTime 0:00:00.0084
Start Address winlogon!WinMainCRTStartup (0x0101cbb0)
Stack Init eb1b0000 Current eb1afcc8 Base eb1b0000 Limit eb1ac000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
eb1afce0 8042d893 00000000 a2034908 00000001 nt!KiSwapThread+0x1b1
eb1afd08 a00019c2 8893bee0 0000000d 00000001 nt!KeWaitForSingleObject+0x1a3
eb1afd44 a0013993 000020ff 00000000 00000001 win32k!xxxSleepThread+0x18a
eb1afd54 a001399f 0006fdd8 80468389 00000000 win32k!xxxWaitMessage+0xe
eb1afd5c 80468389 00000000 00000000 00000000 win32k!NtUserWaitMessage+0xb
eb1afd5c 77e58b53 00000000 00000000 00000000 nt!KiSystemService+0xc9
0006fdd0 77e33630 00000000 00000000 0000ffff USER32!NtUserWaitMessage+0xb
0006fe04 77e44327 000100d2 00000000 00000010 USER32!DialogBox2+0x216
0006fe28 77e38d37 76b90000 76c75c78 00000000 USER32!InternalDialogBox+0xd1
0006fe48 77e39eba 76b90000 76c75c78 00000000 USER32!DialogBoxIndirectParamAorW+0x34
0006fe6c 01011749 76b90000 00000578 00000000 USER32!DialogBoxParamW+0x3d
0006fea8 01018bd3 000755e8 76b90000 00000578 winlogon!TimeoutDialogBoxParam+0x27
0006fee0 76b93701 000755e8 76b90000 00000578 winlogon!WlxDialogBoxParam+0×7b
0006ff08 010164c6 0008d0e0 5ffa0000 000755e8 3rdPartyGINA!WlxDisplaySASNotice+0×43
0006ff20 01014960 000755e8 00000005 00072c9c winlogon!MainLoop+0×96
0006ff58 0101cd06 00071fc8 00000000 00072c9c winlogon!WinMain+0×37a
0006fff4 00000000 7ffdf000 000000c8 00000100 winlogon!WinMainCRTStartup+0×156
THREAD 88980020 Cid 178.188 Teb: 7ffdc000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Alertable
88980108 NotificationTimer
Not impersonating
Owning Process 8893d020
Wait Start TickCount 29930810 Elapsed Ticks: 1652
Context Switch Count 15638
UserTime 0:00:00.0000
KernelTime 0:00:00.0000
Start Address KERNEL32!BaseThreadStartThunk (0x7c57b740)
Win32 Start Address ntdll!RtlpTimerThread (0x77faa02d)
Stack Init bf6f7000 Current bf6f6cc4 Base bf6f7000 Limit bf6f4000 Call 0
Priority 13 BasePriority 13 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
bf6f6cdc 8042d340 bf6f6d64 00bfffac 00bfffac nt!KiSwapThread+0x1b1
bf6f6d04 8052aac9 8046c101 00000001 bf6f6d34 nt!KeDelayExecutionThread+0x182
bf6f6d54 80468389 00000001 00bfffac 00000000 nt!NtDelayExecution+0x7f
bf6f6d54 77f82831 00000001 00bfffac 00000000 nt!KiSystemService+0xc9
00bfff9c 77f842c4 00000001 00bfffac 00000000 ntdll!NtDelayExecution+0xb
00bfffb4 7c57b3bc 0006fe60 00000000 00000000 ntdll!RtlpTimerThread+0x42
00bfffec 00000000 77faa02d 0006fe60 00000000 KERNEL32!BaseThreadStart+0x52
1: kd> dds 0006fee0
0006fee0 0006ff08
0006fee4 76b93701 3rdPartyGINA!WlxDisplaySASNotice+0x43
0006fee8 000755e8
0006feec 76b90000 3rdParty
0006fef0 00000578
0006fef4 00000000
0006fef8 76b9370b 3rdParty!WlxDisplaySASNotice+0x4d
0006fefc 0008d0e0
0006ff00 00000008
0006ff04 00000080
0006ff08 0006ff20
0006ff0c 010164c6 winlogon!MainLoop+0x96
0006ff10 0008d0e0
0006ff14 5ffa0000
0006ff18 000755e8
0006ff1c 00000000
0006ff20 0006ff58
0006ff24 01014960 winlogon!WinMain+0x37a
0006ff28 000755e8
0006ff2c 00000005
0006ff30 00072c9c
0006ff34 00000001
0006ff38 000001bc
0006ff3c 00000005
0006ff40 00000001
0006ff44 0000000d
0006ff48 00000000
0006ff4c 00000000
0006ff50 00000000
0006ff54 0000ffe4
0006ff58 0006fff4
0006ff5c 0101cd06 winlogon!WinMainCRTStartup+0x156
- Dmitry Vostokov @ DumpAnalysis.org -
October 11th, 2007 at 3:28 pm
Sometimes the collection of all stack traces from all threads in the system can disprove or decrease the plausibility of the hypothesis that some module is involved. For one case the customer claimed that the specific driver was involved in server freeze. However there was no such module found in all thread stacks.
November 19th, 2007 at 12:19 pm
We can also filter stacks that belong to processes having the same module name, for example, svchost.exe
http://www.dumpanalysis.org/blog/index.php/2007/11/19/filtering-processes/
January 28th, 2008 at 12:29 pm
[…] need to know normal thread stacks when looking at Stack Trace Collection from kernel and complete memory dumps and trying to spot anomalies. In order to fill this gap I […]
November 21st, 2008 at 8:15 am
[…] pattern cooperation in user dumps coming from x64 environments. Its characteristic feature is stack trace collection that appears to be damaged or corrupt with lots of zeroed call sites and sometimes having UNICODE […]
December 9th, 2008 at 6:19 pm
[…] component. But let’s look a bit deeper inside that crash dump. If we list all thread stacks (stack trace collection) we would see another thread with unhandled exception processing […]
February 11th, 2009 at 3:16 pm
[…] for any response I dumped the process again and renamed the dump file as iexplore3.dmp. Comparing thread stack traces in both dumps showed one difference: the blocked OLE/RPC thread obviously processing some […]
April 14th, 2009 at 4:39 pm
[…] - The Year of Debugging 2010 (0×7DA) - The Year of Dump Analysis When constantly looking at Stack Trace Collections from complete or kernel memory dumps we notice that certain processes are always present and […]
April 16th, 2009 at 12:21 pm
[…] description stated that the system was hanging during its startup so we went further to look at a stack trace collection of all its threads and found the main thread of spooler was hanging for about 23 seconds after 4 […]
April 21st, 2009 at 4:22 pm
[…] example is !process 0 ff WinDbg command to output all processes and their thread stack traces (see Stack Trace Collection pattern for other […]
April 28th, 2009 at 10:23 am
[…] fact, stack traces and their collections are specializations of the more general traces. Another example is Historical Information in […]
May 8th, 2009 at 3:06 pm
[…] !locks and !running WinDbg commands didn’t reveal anything. Therefore we decided to list all threads in the system using !stacks and !process 0 ff commands. The former command gives a birds eye overview of threads […]
June 23rd, 2009 at 11:50 pm
[…] look at stack trace collection and we see dozens of threads running through some 3rd-party component. We do not normally expect […]
June 25th, 2009 at 1:18 am
[…] when browsing through stack trace collection I could spot a thread blocked by a message box, find another hidden exception and from it see […]
July 8th, 2009 at 8:57 pm
[…] look at the stack trace collection to find another exception and we see it indeed on 7th stack […]
August 11th, 2009 at 3:04 pm
[…] was reported that one server was hanging during automated reboot. Stack trace collection shows a few suspended and frozen threads. They all belong to the same […]
September 10th, 2009 at 3:17 pm
[…] it was possible to get stack trace collection using !process 0 ff command with most data from _ETHREAD structures present but most of kernel and […]
October 14th, 2009 at 7:46 pm
[…] list all threads for any exception processing signs and we find one such thread […]
November 12th, 2009 at 5:42 pm
[…] was freezing during user session logoff. A complete memory dump was saved at that time and its stack trace collection (!stacks command) shows the following suspicious thread in a user process (all other threads were […]
November 20th, 2009 at 10:03 am
[…] couldn’t progress and a complete memory dump was saved for later postmortem analysis. !stacks command showed reduced number of waiting threads in one important system […]
November 30th, 2009 at 4:55 pm
[…] user, kernel and complete but considerably longer or shorter stack traces are clearly visible in stack trace collections. I originally planned to call this pattern a Black Swan but after a moment of thought dismissed […]
February 16th, 2010 at 10:34 am
[…] Wait Chain pattern variation is similar to threads waiting for thread objects. When looking at stack trace collection from a complete memory dump file we see several threads in a set of processes are blocked in ALPC […]
April 19th, 2010 at 11:39 pm
[…] critical section list and the address 0162fa04 belongs to the thread #8 (we find it by looking at all thread stacks and search for 0162 in ChildEBP […]
April 26th, 2010 at 7:59 pm
[…] Stack trace collection shows several threads waiting for a critical section when allocating heap blocks or calling loader functions, for example: […]
June 16th, 2010 at 10:40 pm
[…] a culprit component, DllA, or not? Could this blockage result from earlier problems? We search stack trace collection for any other anomalous activity (Semantic Split) and we find indeed a recurrent stack trace […]
July 18th, 2010 at 2:08 pm
[…] was reported that new remote sessions couldn’t be created. A complete memory dump stack trace collection log lists a special process that would not normally be present in a fully initialized session: […]
August 5th, 2010 at 8:07 pm
[…] by the default analysis command (for example, !analyze -v WinDbg command) or by inspecting a stack trace collection. However if we didn’t see any exception thread it doesn’t mean that no exception had […]
October 13th, 2010 at 9:56 pm
[…] dumps have some extension in the dimension of narrativity because it is possible to get stack traces and other execution residue from them that provide partial fragments of a software […]
November 18th, 2011 at 10:54 pm
Added !sprocess command
November 21st, 2011 at 3:06 pm
Added !for_each_thread
October 31st, 2012 at 3:54 pm
For Windows 8 the latest WinDbg 6.2.9200.16384 the parameter ff doesn’t show stack traces so we can use 1f or 16.
November 24th, 2019 at 6:27 pm
Windows 10: !findthreads
January 21st, 2022 at 9:35 pm
~*k 1 lists the first frame only like thread info GDB command.