Crash Dump Analysis Patterns (Part 54)
Thursday, February 28th, 2008Sometimes when listing processes we see the so called Zombie Processes. They are better visible in the output of !vm command as processes with zero private memory:
0: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 999294 ( 3997176 Kb)
Page File: \??\C:\pagefile.sys
Current: 5995520 Kb Free Space: 5324040 Kb
Minimum: 5995520 Kb Maximum: 5995520 Kb
Available Pages: 626415 ( 2505660 Kb)
ResAvail Pages: 902639 ( 3610556 Kb)
Locked IO Pages: 121 ( 484 Kb)
Free System PTEs: 201508 ( 806032 Kb)
Free NP PTEs: 32766 ( 131064 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 256 ( 1024 Kb)
Modified PF Pages: 256 ( 1024 Kb)
NonPagedPool Usage: 12304 ( 49216 Kb)
NonPagedPool Max: 65359 ( 261436 Kb)
PagedPool 0 Usage: 18737 ( 74948 Kb)
PagedPool 1 Usage: 2131 ( 8524 Kb)
PagedPool 2 Usage: 2104 ( 8416 Kb)
PagedPool 3 Usage: 2140 ( 8560 Kb)
PagedPool 4 Usage: 2134 ( 8536 Kb)
PagedPool Usage: 27246 ( 108984 Kb)
PagedPool Maximum: 67072 ( 268288 Kb)
Shared Commit: 60867 ( 243468 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 14359 ( 57436 Kb)
PagedPool Commit: 27300 ( 109200 Kb)
Driver Commit: 1662 ( 6648 Kb)
Committed pages: 501592 ( 2006368 Kb)
Commit limit: 2456879 ( 9827516 Kb)
Total Private: 368810 ( 1475240 Kb)
...
...
...
3654 explorer.exe 2083 ( 8332 Kb)
037c MyService.exe 2082 ( 8328 Kb)
315c explorer.exe 2045 ( 8180 Kb)
…
…
…
0588 svchost.exe 360 ( 1440 Kb)
3f94 csrss.exe 288 ( 1152 Kb)
0acc svchost.exe 245 ( 980 Kb)
0380 smss.exe 38 ( 152 Kb)
0004 System 7 ( 28 Kb)
6ee8 cmd.exe 0 ( 0 Kb)
6d7c cmd.exe 0 ( 0 Kb)
6ca8 cmd.exe 0 ( 0 Kb)
6b48 IEXPLORE.EXE 0 ( 0 Kb)
6ac4 cmd.exe 0 ( 0 Kb)
69e8 cmd.exe 0 ( 0 Kb)
69dc cmd.exe 0 ( 0 Kb)
68dc AcroRd32.exe 0 ( 0 Kb)
6860 cmd.exe 0 ( 0 Kb)
6858 cmd.exe 0 ( 0 Kb)
67d8 cmd.exe 0 ( 0 Kb)
6684 AcroRd32.exe 0 ( 0 Kb)
6484 cmd.exe 0 ( 0 Kb)
6464 cmd.exe 0 ( 0 Kb)
6288 cmd.exe 0 ( 0 Kb)
626c cmd.exe 0 ( 0 Kb)
6260 cmd.exe 0 ( 0 Kb)
6258 cmd.exe 0 ( 0 Kb)
620c IEXPLORE.EXE 0 ( 0 Kb)
60f0 cmd.exe 0 ( 0 Kb)
5fa4 cmd.exe 0 ( 0 Kb)
5f60 cmd.exe 0 ( 0 Kb)
5eec cmd.exe 0 ( 0 Kb)
5d24 IEXPLORE.EXE 0 ( 0 Kb)
5bd4 cmd.exe 0 ( 0 Kb)
5b9c cmd.exe 0 ( 0 Kb)
5b10 cmd.exe 0 ( 0 Kb)
5b08 cmd.exe 0 ( 0 Kb)
5a4c cmd.exe 0 ( 0 Kb)
5a08 cmd.exe 0 ( 0 Kb)
5934 cmd.exe 0 ( 0 Kb)
58b8 cmd.exe 0 ( 0 Kb)
56dc cmd.exe 0 ( 0 Kb)
558c cmd.exe 0 ( 0 Kb)
5588 cmd.exe 0 ( 0 Kb)
5574 cmd.exe 0 ( 0 Kb)
5430 cmd.exe 0 ( 0 Kb)
5424 cmd.exe 0 ( 0 Kb)
53b0 cmd.exe 0 ( 0 Kb)
5174 explorer.exe 0 ( 0 Kb)
5068 cmd.exe 0 ( 0 Kb)
5028 IEXPLORE.EXE 0 ( 0 Kb)
5004 cmd.exe 0 ( 0 Kb)
4f3c javaw.exe 0 ( 0 Kb)
4de4 cmd.exe 0 ( 0 Kb)
4dd8 cmd.exe 0 ( 0 Kb)
4c50 cmd.exe 0 ( 0 Kb)
4c48 cmd.exe 0 ( 0 Kb)
4c08 cmd.exe 0 ( 0 Kb)
4a8c cmd.exe 0 ( 0 Kb)
49ac cmd.exe 0 ( 0 Kb)
4938 cmd.exe 0 ( 0 Kb)
4928 cmd.exe 0 ( 0 Kb)
491c cmd.exe 0 ( 0 Kb)
4868 POWERPNT.EXE 0 ( 0 Kb)
4724 cmd.exe 0 ( 0 Kb)
46cc cmd.exe 0 ( 0 Kb)
44a8 cmd.exe 0 ( 0 Kb)
43cc cmd.exe 0 ( 0 Kb)
4350 cmd.exe 0 ( 0 Kb)
4208 cmd.exe 0 ( 0 Kb)
41f4 cmd.exe 0 ( 0 Kb)
41ec cmd.exe 0 ( 0 Kb)
4170 cmd.exe 0 ( 0 Kb)
40bc cmd.exe 0 ( 0 Kb)
3ddc cmd.exe 0 ( 0 Kb)
3dcc cmd.exe 0 ( 0 Kb)
3db8 cmd.exe 0 ( 0 Kb)
3d88 cmd.exe 0 ( 0 Kb)
3d10 cmd.exe 0 ( 0 Kb)
3cac cmd.exe 0 ( 0 Kb)
3ca4 cmd.exe 0 ( 0 Kb)
3c88 cmd.exe 0 ( 0 Kb)
337c cmd.exe 0 ( 0 Kb)
3310 cmd.exe 0 ( 0 Kb)
3308 cmd.exe 0 ( 0 Kb)
32f0 cmd.exe 0 ( 0 Kb)
32b8 cmd.exe 0 ( 0 Kb)
2ed0 cmd.exe 0 ( 0 Kb)
2eb8 cmd.exe 0 ( 0 Kb)
2e28 cmd.exe 0 ( 0 Kb)
2d44 AcroRd32.exe 0 ( 0 Kb)
2d24 cmd.exe 0 ( 0 Kb)
2c94 cmd.exe 0 ( 0 Kb)
2c54 IEXPLORE.EXE 0 ( 0 Kb)
2a28 cmd.exe 0 ( 0 Kb)
29e4 cmd.exe 0 ( 0 Kb)
2990 cmd.exe 0 ( 0 Kb)
28c0 cmd.exe 0 ( 0 Kb)
25a0 cmd.exe 0 ( 0 Kb)
2558 cmd.exe 0 ( 0 Kb)
2478 cmd.exe 0 ( 0 Kb)
244c cmd.exe 0 ( 0 Kb)
23dc cmd.exe 0 ( 0 Kb)
2320 cmd.exe 0 ( 0 Kb)
2280 cmd.exe 0 ( 0 Kb)
2130 cmd.exe 0 ( 0 Kb)
205c cmd.exe 0 ( 0 Kb)
2014 cmd.exe 0 ( 0 Kb)
1fd8 cmd.exe 0 ( 0 Kb)
1fa0 cmd.exe 0 ( 0 Kb)
1eb8 cmd.exe 0 ( 0 Kb)
1d68 IEXPLORE.EXE 0 ( 0 Kb)
1cb8 cmd.exe 0 ( 0 Kb)
1c9c cmd.exe 0 ( 0 Kb)
1c50 cmd.exe 0 ( 0 Kb)
1a74 cmd.exe 0 ( 0 Kb)
1954 cmd.exe 0 ( 0 Kb)
1948 cmd.exe 0 ( 0 Kb)
06e4 cmd.exe 0 ( 0 Kb)
0650 cmd.exe 0 ( 0 Kb)
We see lots of cmd.exe processes. Let’s examine a few of them:
0: kd> !process 0650
Searching for Process with Cid == 650
PROCESS 89237d88 SessionId: 0 Cid: 0650 Peb: 7ffde000 ParentCid: 037c
DirBase: f3b31940 ObjectTable: 00000000 HandleCount: 0.
Image: cmd.exe
VadRoot 00000000 Vads 0 Clone 0 Private 0. Modified 2. Locked 0.
DeviceMap e10038a8
Token e4eb5b98
ElapsedTime 1 Day 00:16:11.706
UserTime 00:00:00.015
KernelTime 00:00:00.015
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (7, 50, 345) (28KB, 200KB, 1380KB)
PeakWorkingSetSize 588
VirtualSize 11 Mb
PeakVirtualSize 14 Mb
PageFaultCount 663
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 0
No active threads
0: kd> !process 2130
Searching for Process with Cid == 2130
PROCESS 89648020 SessionId: 0 Cid: 2130 Peb: 7ffdc000 ParentCid: 037c
DirBase: f3b31060 ObjectTable: 00000000 HandleCount: 0.
Image: cmd.exe
VadRoot 00000000 Vads 0 Clone 0 Private 0. Modified 2. Locked 0.
DeviceMap e10038a8
Token e5167bb8
ElapsedTime 15:40:17.643
UserTime 00:00:00.015
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (7, 50, 345) (28KB, 200KB, 1380KB)
PeakWorkingSetSize 545
VirtualSize 11 Mb
PeakVirtualSize 14 Mb
PageFaultCount 621
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 0
No active threads
We see that most of them have Parent PID as 037c which is MyService.exe. Let’s peek inside its handle table:
0: kd> !kdexts.handle 0 3 037c
processor number 0, process 0000037c
Searching for Process with Cid == 37c
PROCESS 8a8fa8c0 SessionId: 0 Cid: 037c Peb: 7ffd8000 ParentCid: 04ac
DirBase: f3b10360 ObjectTable: e1c276b8 HandleCount: 500.
Image: MyService.exe
Handle table at e272d000 with 500 Entries in use
0004: Object: e1000638 GrantedAccess: 00000003 Entry: e1caf008
Object: e1000638 Type: (8ad79ad0) KeyedEvent
ObjectHeader: e1000620 (old version)
HandleCount: 151 PointerCount: 152
Directory Object: e1001898 Name: CritSecOutOfMemoryEvent
0008: Object: 8a8cfdf8 GrantedAccess: 001f0003 Entry: e1caf010
Object: 8a8cfdf8 Type: (8ad7a990) Event
ObjectHeader: 8a8cfde0 (old version)
HandleCount: 1 PointerCount: 1
000c: Object: e186d690 GrantedAccess: 00000003 Entry: e1caf018
Object: e186d690 Type: (8ad84e70) Directory
ObjectHeader: e186d678 (old version)
HandleCount: 150 PointerCount: 181
Directory Object: e1003b28 Name: KnownDlls
0010: Object: 8a8d1328 GrantedAccess: 00100020 (Inherit) Entry: e1caf020
Object: 8a8d1328 Type: (8ad74900) File
ObjectHeader: 8a8d1310 (old version)
HandleCount: 1 PointerCount: 1
Directory Object: 00000000 Name: \WINDOWS\system32 {HarddiskVolume1}
...
...
...
0484: Object: 89648020 GrantedAccess: 001f0fff Entry: e1caf908
Object: 89648020 Type: (8ad84900) Process
ObjectHeader: 89648008 (old version)
HandleCount: 1 PointerCount: 2
…
…
…
0510: Object: 89237d88 GrantedAccess: 001f0fff Entry: e1cafa20
Object: 89237d88 Type: (8ad84900) Process
ObjectHeader: 89237d70 (old version)
HandleCount: 1 PointerCount: 2
…
…
…
Therefore we may guess that MyService.exe probably forgot to close process handles either after launching cmd.exe or after waiting for their exit when process objects become signaled:
0510: Object: 89237d88 GrantedAccess: 001f0fff Entry: e1cafa20
Object: 89237d88 Type: (8ad84900) Process
ObjectHeader: 89237d70 (old version)
HandleCount: 1 PointerCount: 2
0: kd> dt _DISPATCHER_HEADER 89237d88
ntdll!_DISPATCHER_HEADER
+0x000 Type : 0x3 '' ; PROCESS OBJECT
+0x001 Absolute : 0 ''
+0x001 NpxIrql : 0 ''
+0x002 Size : 0x1e ''
+0x002 Hand : 0x1e ''
+0x003 Inserted : 0 ''
+0x003 DebugActive : 0 ''
+0x000 Lock : 1966083
+0×004 SignalState : 1
+0×008 WaitListHead : _LIST_ENTRY [ 0×89237d90 - 0×89237d90 ]
This pattern can also be seen a specialization of a more general Handle Leak pattern.
- Dmitry Vostokov @ DumpAnalysis.org -