Software Diagnostics Library

I remember in my old days of PDP-11 system programming I learnt about the system call to spawn processes and wrote a program in assembly language that was spawning itself. This recursive spawning resulted in geometrical progression of running tasks and brought RSX-11M system to halt very quickly. Recently I observed the similar but non-recursive Process Factory pattern in one of memory dumps: explorer was relentlessly creating application.exe processes and at the time some effect was noticed there were more than 5,000 of them:

1: kd> !vm
[...]
5d20 application.exe 212 ( 848 Kb)
5d08 application.exe 212 ( 848 Kb)
5d04 application.exe 212 ( 848 Kb)
5cf8 application.exe 212 ( 848 Kb)
5cf0 application.exe 212 ( 848 Kb)
5ce8 application.exe 212 ( 848 Kb)
5cdc application.exe 212 ( 848 Kb)
5ccc application.exe 212 ( 848 Kb)
5cc8 application.exe 212 ( 848 Kb)
5cc0 application.exe 212 ( 848 Kb)
5ca8 application.exe 212 ( 848 Kb)
5c9c application.exe 212 ( 848 Kb)
5c98 application.exe 212 ( 848 Kb)
5c90 application.exe 212 ( 848 Kb)
5c88 application.exe 212 ( 848 Kb)
5c7c application.exe 212 ( 848 Kb)
5c70 application.exe 212 ( 848 Kb)
5c68 application.exe 212 ( 848 Kb)
5c64 application.exe 212 ( 848 Kb)
5c60 application.exe 212 ( 848 Kb)
5c50 application.exe 212 ( 848 Kb)
5c4c application.exe 212 ( 848 Kb)
5c44 application.exe 212 ( 848 Kb)
5c3c application.exe 212 ( 848 Kb)
5c34 application.exe 212 ( 848 Kb)
5c2c application.exe 212 ( 848 Kb)
5c24 application.exe 212 ( 848 Kb)
5c1c application.exe 212 ( 848 Kb)
5bf8 application.exe 212 ( 848 Kb)
5be0 application.exe 212 ( 848 Kb)
5bd4 application.exe 212 ( 848 Kb)
5bd0 application.exe 212 ( 848 Kb)
5ba4 application.exe 212 ( 848 Kb)
5b58 application.exe 212 ( 848 Kb)
5b50 application.exe 212 ( 848 Kb)
5b44 application.exe 212 ( 848 Kb)
5b38 application.exe 212 ( 848 Kb)
5b30 application.exe 212 ( 848 Kb)
5b04 application.exe 212 ( 848 Kb)
5af4 application.exe 212 ( 848 Kb)
5ad8 application.exe 212 ( 848 Kb)
5ad4 application.exe 212 ( 848 Kb)
5ac8 application.exe 212 ( 848 Kb)
5ac4 application.exe 212 ( 848 Kb)
5ab4 application.exe 212 ( 848 Kb)
5aa4 application.exe 212 ( 848 Kb)
5a9c application.exe 212 ( 848 Kb)
5a94 application.exe 212 ( 848 Kb)
5a8c application.exe 212 ( 848 Kb)
5a88 application.exe 212 ( 848 Kb)
5a74 application.exe 212 ( 848 Kb)
[...]

1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8b57f020  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: cffb3020  ObjectTable: e1003da0  HandleCount: 3932.
    Image: System

PROCESS 8a9f8d88  SessionId: none  Cid: 01b8    Peb: 7ffdf000  ParentCid: 0004
    DirBase: cffb3040  ObjectTable: e13e3f68  HandleCount: 111.
    Image: smss.exe

PROCESS 89f0d508  SessionId: 0  Cid: 01f0    Peb: 7ffd8000  ParentCid: 01b8
    DirBase: cffb3060  ObjectTable: e16bc370  HandleCount: 1292.
    Image: csrss.exe

PROCESS 89eea7c8  SessionId: 0  Cid: 0208    Peb: 7ffde000  ParentCid: 01b8
    DirBase: cffb3080  ObjectTable: e14b4160  HandleCount: 564.
    Image: winlogon.exe

[...]

PROCESS 8607c020  SessionId: 1  Cid: 44c8    Peb: 7ffdc000  ParentCid: 4cf8
    DirBase: cffb7080  ObjectTable: e3c9fd38  HandleCount: 25407.
    Image: explorer.exe

[...]

PROCESS 85e1d020  SessionId: 1  Cid: 538c    Peb: 7ffda000  ParentCid: 44c8
    DirBase: cffb8980  ObjectTable: e8065b20  HandleCount:  39.
    Image: application.exe

PROCESS 85c74610  SessionId: 1  Cid: 5394    Peb: 7ffd9000  ParentCid: 44c8
    DirBase: cffb89a0  ObjectTable: e6951878  HandleCount:  39.
    Image: application.exe

PROCESS 85c81020  SessionId: 1  Cid: 53a4    Peb: 7ffd7000  ParentCid: 44c8
    DirBase: cffb89c0  ObjectTable: e6d2f600  HandleCount:  39.
    Image: application.exe

PROCESS 85c6fb18  SessionId: 1  Cid: 53a8    Peb: 7ffd7000  ParentCid: 44c8
    DirBase: cffb89e0  ObjectTable: e54df078  HandleCount:  39.
    Image: application.exe

PROCESS 85c60020  SessionId: 1  Cid: 53bc    Peb: 7ffdf000  ParentCid: 44c8
    DirBase: cffb8a40  ObjectTable: e1214e90  HandleCount:  39.
    Image: application.exe

PROCESS 85c5d380  SessionId: 1  Cid: 53c8    Peb: 7ffde000  ParentCid: 44c8
    DirBase: cffb8a60  ObjectTable: e7baf638  HandleCount:  39.
    Image: application.exe

PROCESS 85c648b8  SessionId: 1  Cid: 53dc    Peb: 7ffde000  ParentCid: 44c8
    DirBase: cffb8a80  ObjectTable: e759d060  HandleCount:  39.
    Image: application.exe

PROCESS 85c62528  SessionId: 1  Cid: 53e0    Peb: 7ffde000  ParentCid: 44c8
    DirBase: cffb8aa0  ObjectTable: e3b8fa00  HandleCount:  39.
    Image: application.exe

PROCESS 85c59d88  SessionId: 1  Cid: 53e8    Peb: 7ffdc000  ParentCid: 44c8
    DirBase: cffb8ac0  ObjectTable: e31751e0  HandleCount:  39.
    Image: application.exe

PROCESS 85c46d88  SessionId: 1  Cid: 542c    Peb: 7ffd5000  ParentCid: 4d9c
    DirBase: cffb8b00  ObjectTable: e6fbc500  HandleCount: 136.
    Image: nlapplication.exe

PROCESS 85c3c020  SessionId: 1  Cid: 5464    Peb: 7ffdc000  ParentCid: 44c8
    DirBase: cffb8b40  ObjectTable: e218b948  HandleCount:  39.
    Image: application.exe

PROCESS 85c2a020  SessionId: 1  Cid: 546c    Peb: 7ffdb000  ParentCid: 44c8
    DirBase: cffb8b60  ObjectTable: e639a8d0  HandleCount:  39.
    Image: application.exe

PROCESS 85c202c8  SessionId: 1  Cid: 5474    Peb: 7ffd7000  ParentCid: 44c8
    DirBase: cffb8b80  ObjectTable: e517caa8  HandleCount:  39.
    Image: application.exe

PROCESS 85c1b020  SessionId: 1  Cid: 547c    Peb: 7ffd6000  ParentCid: 44c8
    DirBase: cffb8ba0  ObjectTable: e6c0cbc0  HandleCount:  39.
    Image: application.exe

PROCESS 85c1dd88  SessionId: 1  Cid: 5484    Peb: 7ffd5000  ParentCid: 44c8
    DirBase: cffb8bc0  ObjectTable: e4a42f68  HandleCount:  39.
    Image: application.exe

PROCESS 85d3ed88  SessionId: 1  Cid: 5488    Peb: 7ffd5000  ParentCid: 44c8
    DirBase: cffb8be0  ObjectTable: e68558f0  HandleCount:  39.
    Image: application.exe

[...]

We see that all created processes have the same parent process with PID 44c8 and when we inspect it we see many threads inside creating application.exe process:

1: kd> .process /r /p 8607c020
Implicit process is now 8607c020
Loading User Symbols

1: kd> !process 8607c020
PROCESS 8607c020  SessionId: 1  Cid: 44c8    Peb: 7ffdc000  ParentCid: 4cf8
    DirBase: cffb7080  ObjectTable: e3c9fd38  HandleCount: 25407.
    Image: explorer.exe
    VadRoot 88efec98 Vads 3445 Clone 0 Private 30423. Modified 71292. Locked 0.
    DeviceMap e3743340
    Token                             e29be5e0
    ElapsedTime                       00:54:31.359
    UserTime                          00:00:19.234
    KernelTime                        00:04:04.828
    QuotaPoolUsage[PagedPool]         1075132
    QuotaPoolUsage[NonPagedPool]      137800
    Working Set Sizes (now,min,max)  (15457, 50, 345) (61828KB, 200KB, 1380KB)
    PeakWorkingSetSize                48919
    VirtualSize                       585 Mb
    PeakVirtualSize                   978 Mb
    PageFaultCount                    123488
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      49919

[...]

THREAD 84f25300  Cid 44c8.6288  Teb: 7ff8e000 Win32Thread: bc486830 READY
IRP List:
    88699110: (0006,0220) Flags: 00000884  Mdl: 00000000
Not impersonating
DeviceMap                 e3743340
Owning Process            8607c020       Image:         explorer.exe
Wait Start TickCount      1327981        Ticks: 29 (0:00:00:00.453)
Context Switch Count      145332                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.093
Win32 Start Address SHLWAPI!SHCreateThread (0x77ec3ea5)
Start Address kernel32!BaseThreadStartThunk (0x7c8217ec)
Stack Init a98e4000 Current a98e3700 Base a98e4000 Limit a98e0000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
a98e3718 80833ec5 nt!KiSwapContext+0x26
a98e3744 80829bc0 nt!KiSwapThread+0x2e5
a98e378c 8087e0d8 nt!KeWaitForSingleObject+0x346
a98e37c4 8087e397 nt!ExpWaitForResource+0x30
a98e37e4 badff32a nt!ExAcquireResourceExclusiveLite+0x8d
a98e3808 badffe35 driverA+0x132a
a98e3824 bae00208 driverA+0x1e35
a98e3868 bae0e45a driverA+0x2208
a98e38a0 8081e095 driverA+0x1045a
a98e38b4 b972c73b nt!IofCallDriver+0x45
[...]
a98e38e8 b9b194e1 nt!IofCallDriver+0x45
[...]
a98e3940 b85cbf08 nt!IofCallDriver+0x45
a98e3968 b85bcfcc driverB!LowerDevicePassThrough+0x48
a98e398c b85bd63d driverB+0x6fcc
a98e3a24 b85cb167 driverB+0x763d
a98e3a34 b85cb1b7 driverB+0x15167
a98e3a5c 8081e095 driverB!DispatchPassThrough+0x48
a98e3a70 808fb13b nt!IofCallDriver+0x45
a98e3b58 80939c6a nt!IopParseDevice+0xa35
a98e3bd8 80935d9e nt!ObpLookupObjectName+0x5b0
a98e3c2c 808ece57 nt!ObOpenObjectByName+0xea
a98e3ca8 808ee0f1 nt!IopCreateFile+0x447
a98e3d04 808f1e31 nt!IoCreateFile+0xa3
a98e3d44 8088ad3c nt!NtOpenFile+0x27
a98e3d44 7c9485ec nt!KiFastCallEntry+0xfc (TrapFrame @ a98e3d64)
03bbda04 7c82bdf6 ntdll!KiFastSystemCallRet
03bbda2c 7c82dd9a kernel32!BasepSxsCreateStreams+0xe2
03bbda9c 7c82d895 kernel32!BasepSxsCreateProcessCsrMessage+0x136
03bbe2c4 7c8024a0 kernel32!CreateProcessInternalW+0x1943
03bbe2fc 7ca36750 kernel32!CreateProcessW+0×2c
03bbed80 7ca36b45 SHELL32!_SHCreateProcess+0×387
03bbedd4 7ca3617b SHELL32!CShellExecute::_DoExecCommand+0xb4
03bbede0 7ca35a76 SHELL32!CShellExecute::_TryInvokeApplication+0×49
03bbedf4 7ca3599f SHELL32!CShellExecute::ExecuteNormal+0xb1
03bbee08 7ca35933 SHELL32!ShellExecuteNormal+0×30
03bbee24 7ca452ff SHELL32!ShellExecuteExW+0×8d

1: kd> .thread 84e6a600
Implicit thread is now 84e6a600

1: kd> kv 100
[...]
03bbda04 7c82bdf6 001200a9 03bbda8c 03bbdb20 ntdll!KiFastSystemCallRet
03bbda2c 7c82dd9a 00000000 00000003 001200a9 kernel32!BasepSxsCreateStreams+0xe2
03bbda9c 7c82d895 00000000 00000000 03bbdc38 kernel32!BasepSxsCreateProcessCsrMessage+0x136
03bbe2c4 7c8024a0 00000000 01dafb9c 01dad904 kernel32!CreateProcessInternalW+0x1943
03bbe2fc 7ca36750 01dafb9c 01dad904 00000000 kernel32!CreateProcessW+0×2c
03bbed80 7ca36b45 00010098 00000000 01daffac SHELL32!_SHCreateProcess+0×387
[…]

1: kd> du /c 100 01dafb9c
01dafb9c  “C:\Program Files\App Package\Application.exe”

The difference of this pattern and similar Handle Leak or Zombie Processes is the fact that leaks usually happen when a process forgets to close handles but Process Factory creates active processes which are full resource containers and consume system resources, for example, they all have full handle table or consume GDI resources if they are GUI processes.

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Friday, February 13th, 2009 at 3:19 pm and is filed under Crash Dump Analysis, Crash Dump Patterns, Debugging, History. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.