Crash Dump Analysis Patterns (Part 27e)

This is another variant of Stack Trace Collection pattern that shows stack traces from threads currently execution on all CPUs. Although we can see the non-idle running threads from the stack traces corresponding to all processes and their threads we may also want to see idle thread stack traces too. Also, the corresponding WinDbg command (!running -t -i) is faster if we want to double check the output of !analyze -v command in case of BSOD. The latter command may show the stack trace from the current CPU instead of the stack trace from the thread running on a different CPU that caused a bugcheck. Here’s an example from one of the memory dumps for which !analyze -v command shows an incorrect stack trace in the output when we open the dump file. It reports the stack trace from CPU 0 but the bugcheck  happened on CPU 1:

0: kd> !running -t -i

System Processors:  (00000000000000ff)
Idle Processors:  (00000000000000fd)

Prcbs             Current         (pri) Next            (pri) Idle
0    fffff801e5d85180  fffff801e5ddea00 ( 0)                       fffff801e5ddea00  ................

Child-SP          RetAddr           Call Site
fffff801`e8c9eb60 fffff801`e5c69b74 hal!KeQueryPerformanceCounter+0x75
fffff801`e8c9eba0 fffff801`e5c69e01 nt!KiCheckStall+0x2c
fffff801`e8c9ebd0 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
fffff801`e8c9ece0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
fffff801`e8c9ed30 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
fffff801`e8c9ee70 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
fffff801`e8c8c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
fffff801`e8c8c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
fffff801`e8c8c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
fffff801`e8c8cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
fffff801`e8c8cc60 00000000`00000000 nt!KiIdleLoop+0x2c

1    ffffd000f0975180  ffffe0000d726880 (12)                       ffffd000f09813c0  …………….

Child-SP          RetAddr           Call Site
ffffd000`202cb618 fffff801`e5a1cc3c hal!HalpAcpiPmRegisterReadPort+0x1b
ffffd000`202cb620 fffff801`e5a417e7 hal!HalpAcpiPmRegisterRead+0x30
ffffd000`202cb650 fffff801`e5c66af5 hal!HaliHaltSystem+0x53
ffffd000`202cb690 fffff801`e5c66741 nt!KiBugCheckDebugBreak+0×99
ffffd000`202cb6f0 fffff801`e5bd2aa4 nt!KeBugCheck2+0xc6d
ffffd000`202cbe00 fffff801`e5bde4e9 nt!KeBugCheckEx+0×104
ffffd000`202cbe40 fffff801`e5bdcd3a nt!KiBugCheckDispatch+0×69
ffffd000`202cbf80 fffff800`913601da nt!KiPageFault+0×23a
ffffd000`202cc118 fffff800`91363710 DriverA!memcpy+0×21a

[…]

2    ffffd000f09ee180  ffffd000f09fa3c0 ( 0)                       ffffd000f09fa3c0  ................

Child-SP          RetAddr           Call Site
ffffd000`f09f9f88 fffff801`e5c69e01 nt!KiCheckStall+0xa
ffffd000`f09f9f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`f09fa0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`f09fa0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`f09fa230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb5938e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb5938f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb593920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb593b10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb593c60 00000000`00000000 nt!KiIdleLoop+0x2c

3    ffffd000eb5e5180  ffffd000eb5f13c0 ( 0)                       ffffd000eb5f13c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb5f0f60 fffff801`e5c69e01 nt!KiCheckStall+0x5f
ffffd000`eb5f0f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb5f10a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb5f10f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb5f1230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb5fa8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb5fa8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb5fa920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb5fab10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb5fac60 00000000`00000000 nt!KiIdleLoop+0x2c

4    ffffd000f08d1180  ffffd000f08dd3c0 ( 0)                       ffffd000f08dd3c0  ................

Child-SP          RetAddr           Call Site
ffffd000`f08dcf90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x227
ffffd000`f08dd0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`f08dd0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`f08dd230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb85b8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb85b8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb85b920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb85bb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb85bc60 00000000`00000000 nt!KiIdleLoop+0x2c

5    ffffd000eb8ad180  ffffd000eb8b93c0 ( 0)                       ffffd000eb8b93c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb8b8f60 fffff801`e5c69e01 nt!KiCheckStall+0x75
ffffd000`eb8b8f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb8b90a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb8b90f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb8b9230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb8db8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb8db8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb8db920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb8dbb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb8dbc60 00000000`00000000 nt!KiIdleLoop+0x2c

6    ffffd000eb92a180  ffffd000eb9363c0 ( 0)                       ffffd000eb9363c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb935f60 fffff801`e5c69e01 nt!KiCheckStall+0x75
ffffd000`eb935f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb9360a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb9360f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb936230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb93f8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb93f8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb93f920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb93fb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb93fc60 00000000`00000000 nt!KiIdleLoop+0x2c

7    ffffd000eb967180  ffffd000eb9733c0 ( 0)                       ffffd000eb9733c0  ................

Child-SP          RetAddr           Call Site
ffffd000`eb972f60 fffff801`e5c69e01 nt!KiCheckStall+0x75
ffffd000`eb972f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231
ffffd000`eb9730a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b
ffffd000`eb9730f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82
ffffd000`eb973230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176
ffffd000`eb97c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf
ffffd000`eb97c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa
ffffd000`eb97c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8
ffffd000`eb97cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6
ffffd000`eb97cc60 00000000`00000000 nt!KiIdleLoop+0x2c

This command is obviously faster than repeatedly switching to subsequent CPUs using ~s command and then checking the corresponding stack trace (k). It also helps in diagnosing Spiking Threads in kernel and complete memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One Response to “Crash Dump Analysis Patterns (Part 27e)”

  1. Dmitry Vostokov Says:

    .trap command may be incompatible with !running -t command: the latter shows all CPU threads having the same stack trace as in the output of k after .trap. (WinDbg 10.0.16299.15)

Leave a Reply