Truncated dump, stack trace collection, waiting thread time and wait chains: pattern cooperation
It was reported that functionality of some applications was not available until ServiceA was killed and restarted. Complete memory dump was forced. For that memory dump WinDbg !stacks command shows every thread bugchecking the system, for example:
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[8ca72020 ApplicationA.exe]
4bc.0004c4 8bf76af0 0000cc4 Blocked nt!KeBugCheckEx+0x1b
4bc.0004d0 8bf53650 000055d Blocked nt!KeBugCheckEx+0x1b
4bc.0004e8 8bf33c88 0000094 Blocked nt!KeBugCheckEx+0x1b
4bc.001ffc 8ba86020 0000eed Blocked nt!KeBugCheckEx+0x1b
4bc.001e30 8b7a0ca8 00003ac Blocked nt!KeBugCheckEx+0x1b
4bc.001934 8b664020 0000eed Blocked nt!KeBugCheckEx+0x1b
4bc.001af0 8ca6d3b0 0000094 Blocked nt!KeBugCheckEx+0x1b
4bc.001cac 8bf41af0 0001412 Blocked nt!KeBugCheckEx+0x1b
4bc.00141c 8b6458d0 0000094 Blocked nt!KeBugCheckEx+0x1b
[...]
Seeing this nonsense I checked that complete dump was truncated by half because page file was 4Gb but the amount of physical memory was 8Gb:
0: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 2096553 ( 8386212 Kb)
Paging File Name paged out
Current: 4195328 Kb Free Space: 4182336 Kb
Minimum: 4195328 Kb Maximum: 4195328 Kb
[…]
However it was possible to get stack trace collection using !process 0 ff command with most data from _ETHREAD structures present but most of kernel and user space stack traces not present in the output. ServiceA has many threads waiting for LPC requests during last 5 minutes (the bugcheck was forced after 3 - 4 minutes the issue manifested itself):
THREAD 8cc9b590 Cid 053c.0668 Teb: 7ffaf000 Win32Thread: b84e5770 WAIT: (Unknown) UserMode Non-Alertable
8cc9b77c Semaphore Limit 0x1
Waiting for reply to LPC MessageId 0005bbdf:
Current LPC port e2cee338
IRP List:
8b6548b8: (0006,0268) Flags: 00000000 Mdl: 00000000
Not impersonating
DeviceMap e1003860
Owning Process 8bf81d88 Image: ServiceA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 193823 Ticks: 13752 (0:00:03:34.875)
Context Switch Count 389 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0×7d1f5e70
Start Address 0×77e617ec
Stack Init b3a44000 Current b3a43c08 Base b3a44000 Limit b3a41000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
The similar waits were found for ApplicationB and ApplicationC processes. Therefore it was advised to limit the amount of physical memory to 4Gb and reproduce the issue or if the liveliness of the system needs to be preserved to manually dump the following processes on the next occasion:
ServiceA.exe
ApplicationB.exe
ApplicationC.exe
- Dmitry Vostokov @ DumpAnalysis.org -