Case Study: Extremely Inconsitent Dump and CPU Spike

100% CPU consumption was reported for one system and a complete memory dump was generated. Unfortunately, it was very inconsistent:

0: kd> !process 0 0
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
GetContextState failed, 0xD0000147
Unable to read selector for PCR for processor 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8b57f648  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: bffd0020  ObjectTable: e1000e10  HandleCount: 3801.
    Image: System

[...]

PROCESS 8a33fd88  SessionId: 4294963440  Cid: 1508    Peb: 7ffdb000  ParentCid: 3a74
    DirBase: bffd2760  ObjectTable: e653c110  HandleCount: 1664628019.
    Image: explorer.exe

[...]

PROCESS 87bd9d88  SessionId: 4294963440  Cid: 3088    Peb: 7ffda000  ParentCid: 1508
    DirBase: bffd23e0  ObjectTable: e4e73d30  HandleCount: 1717711416.
    Image: iexplore.exe

[...]

PROCESS 88c741a0  SessionId: 0  Cid: 46b0    Peb: 7ffd9000  ParentCid: 01f8
    DirBase: bffd2ac0  ObjectTable: e8b60c58  HandleCount: 1415935346.
    Image: csrss.exe

[...]

!process 0 ff command was looping through the same system thread forever. Fortunately !running command was functional:

0: kd> !running
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter

System Processors 3 (affinity mask)
  Idle Processors 0

Prcbs  Current   Next   
  0    ffdff120  888ab360            …………….
  1    f7727120  880d1db0            …………….

GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147

Curiously !ready command showed a different thread running on the same processor 0 before infinitely looping (it was aborted):

0: kd> !ready
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
Processor 0: Ready Threads at priority 6
    THREAD 88fe2b30  Cid 3b8c.232c  Teb: 7ffdf000 Win32Thread: bc6b38f0 RUNNING on processor 0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
[…]

The both “running” threads were showing signs of a spiking thread:

0: kd> !thread 88fe2b30 1f
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
THREAD 88fe2b30  Cid 3b8c.232c  Teb: 7ffdf000 Win32Thread: bc6b38f0 RUNNING on processor 0
Not impersonating
DeviceMap                 e3899900
Owning Process            8862ead8       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      1980369     
Context Switch Count      239076                 LargeStack
UserTime                  00:01:33.187
KernelTime                00:01:49.734

Win32 Start Address 0×0066c181
Start Address 0×77e617f8
Stack Init b97bfbd0 Current b97bf85c Base b97c0000 Limit b97b9000 Call b97bfbd8
Priority 8 BasePriority 8 PriorityDecrement 0
Unable to get context for thread running on processor 0, HRESULT 0×80004002

GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002

0: kd> !thread 888ab360 1f
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
THREAD 888ab360  Cid 2a3c.4260  Teb: 7ffde000 Win32Thread: bc190570 WAIT: (Unknown) UserMode Non-Alertable
    88e4d8d8  SynchronizationEvent
Not impersonating
DeviceMap                 e62a50e0
Owning Process            8a1a5d88       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      1979505     
Context Switch Count      167668                 LargeStack
UserTime                  00:01:03.468
KernelTime                00:01:21.875

Win32 Start Address ApplicationA (0×0066c181)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ba884000 Current ba883bac Base ba884000 Limit ba87d000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
ChildEBP RetAddr 
ba883c14 bf8a1305 win32k!RGNOBJ::UpdateUserRgn+0×5d
ba883c38 bf8a2a1a win32k!xxxSendMessage+0×1b
ba883c64 bf8a2ac3 win32k!xxxUpdateWindow2+0×79
ba883c84 bf8a1a6a win32k!xxxInternalUpdateWindow+0×6f
ba883cc8 bf8a291b win32k!xxxInternalInvalidate+0×148
ba883cf4 bf858314 win32k!xxxRedrawWindow+0×103
ba883d4c 8088b41c win32k!NtUserRedrawWindow+0xac
ba883d4c 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ ba883d64)
0012fd10 7739b82a ntdll!KiFastSystemCallRet
0012fd98 78a3ed73 USER32!UserCallWinProcCheckWow+0×5c
0012fdb8 78a3f68b mfc90u!CWnd::DefWindowProcW+0×44
0012fdd4 78a3e29a mfc90u!CWnd::WindowProc+0×3b
0012fe58 78585f1a mfc90u!AfxCallWndProc+0xa3
7739ceb8 c25d008b MSVCR90!_msize+0xf8
WARNING: Frame IP not in any known module. Following frames may be wrong.
7739cec0 9090f8eb 0xc25d008b
7739cec4 8b909090 0×9090f8eb
7739cec8 ec8b55ff 0×8b909090
7739cecc e8084d8b 0xec8b55ff
7739ced0 ffffe838 0xe8084d8b
7739ced4 0374c085 0xffffe838
7739ced8 5d78408b 0×374c085
7739cedc 900004c2 0×5d78408b
7739cee0 90909090 0×900004c2
7739cee4 8b55ff8b 0×90909090
7739cee8 18a164ec 0×8b55ff8b
7739ceec 83000000 0×18a164ec
7739cef0 0f004078 0×83000000
7739cef4 fe87d484 0xf004078
7739cef8 087d83ff 0xfe87d484
7739cefc 2c830f20 0×87d83ff
7739cf00 64ffff94 0×2c830f20
7739cf04 0018158b 0×64ffff94
7739cf08 828b0000 0×18158b
7739cf0c 00000000 0×828b0000

GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147

We see that both threads belong to 2 process instances of ApplicationA.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One Response to “Case Study: Extremely Inconsitent Dump and CPU Spike”

  1. Takashi Toyota Says:

    Hi,

    As always, this is an interesting case.
    Thank you for the info!

    I try to start this type of analysis in the following manner;

    http://www.ttoyota.com/cppdesigner/mylog.php

    Hope it helps.

Leave a Reply

You must be logged in to post a comment.