Crash Dump Analysis Patterns (Part 48)
Special Stack Trace pattern is about stack traces not present in normal crash dumps. The similar pattern is called Special Process which is about processes not running during normal operation or highly domain specific processes that are the sign of certain software environment, for example, OS running inside VMWare or VirtualPC. Here I’ll show one example when identifying specific process led to successful problem identification inside a complete memory dump.
Inspection of running processes shows the presence of Dr. Watson:
5: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 89d9b648 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 54d5d020 ObjectTable: e1000e20 HandleCount: 1711.
Image: System
PROCESS 8979b758 SessionId: none Cid: 01b0 Peb: 7ffdd000 ParentCid: 0004
DirBase: 54d5d040 ObjectTable: e181d8b0 HandleCount: 29.
Image: smss.exe
PROCESS 89793cf0 SessionId: 0 Cid: 01e0 Peb: 7ffde000 ParentCid: 01b0
DirBase: 54d5d060 ObjectTable: e13eea10 HandleCount: 1090.
Image: csrss.exe
...
...
...
PROCESS 8797a600 SessionId: 1 Cid: 17d0 Peb: 7ffdc000 ParentCid: 1720
DirBase: 54d5d8c0 ObjectTable: e2870af8 HandleCount: 243.
Image: explorer.exe
PROCESS 87966d88 SessionId: 2 Cid: 0df0 Peb: 7ffd4000 ParentCid: 01b0
DirBase: 54d5d860 ObjectTable: e284cd48 HandleCount: 53.
Image: csrss.exe
PROCESS 879767c8 SessionId: 0 Cid: 0578 Peb: 7ffde000 ParentCid: 0ca8
DirBase: 54d5d8a0 ObjectTable: e2c05268 HandleCount: 180.
Image: drwtsn32.exe
Inspecting stack traces shows that drwtsn32.exe is waiting for a debugger event so there must be some debugging target (debuggee):
5: kd> .process /r /p 879767c8
Implicit process is now 879767c8
Loading User Symbols
5: kd> !process 879767c8
PROCESS 879767c8 SessionId: 0 Cid: 0578 Peb: 7ffde000 ParentCid: 0ca8
DirBase: 54d5d8a0 ObjectTable: e2c05268 HandleCount: 180.
Image: drwtsn32.exe
VadRoot 88a33cd0 Vads 59 Clone 0 Private 1737. Modified 10792. Locked 0.
DeviceMap e10028e8
Token e2ee2330
ElapsedTime 00:01:12.703
UserTime 00:00:00.203
KernelTime 00:00:00.031
QuotaPoolUsage[PagedPool] 52092
QuotaPoolUsage[NonPagedPool] 2360
Working Set Sizes (now,min,max) (2488, 50, 345) (9952KB, 200KB, 1380KB)
PeakWorkingSetSize 2534
VirtualSize 34 Mb
PeakVirtualSize 38 Mb
PageFaultCount 13685
MemoryPriority BACKGROUND
BasePriority 6
CommitCharge 1927
THREAD 87976250 Cid 0578.04bc Teb: 7ffdd000 Win32Thread: bc14a008 WAIT: (Unknown) UserMode Non-Alertable
87976558 Thread
Not impersonating
DeviceMap e10028e8
Owning Process 879767c8 Image: drwtsn32.exe
Wait Start TickCount 13460 Ticks: 4651 (0:00:01:12.671)
Context Switch Count 15 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address drwtsn32!mainCRTStartup (0x01007c1d)
Start Address kernel32!BaseProcessStartThunk (0x7c8217f8)
Stack Init f433b000 Current f433ac60 Base f433b000 Limit f4337000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr
f433ac78 80833465 nt!KiSwapContext+0x26
f433aca4 80829a62 nt!KiSwapThread+0x2e5
f433acec 80938d0c nt!KeWaitForSingleObject+0x346
f433ad50 8088978c nt!NtWaitForSingleObject+0x9a
f433ad50 7c9485ec nt!KiFastCallEntry+0xfc
0007fe98 7c821c8d ntdll!KiFastSystemCallRet
0007feac 01005557 kernel32!WaitForSingleObject+0x12
0007ff0c 01003ff8 drwtsn32!NotifyWinMain+0x1ba
0007ff44 01007d4c drwtsn32!main+0x31
0007ffc0 7c82f23b drwtsn32!mainCRTStartup+0x12f
0007fff0 00000000 kernel32!BaseProcessStart+0x23
THREAD 87976558 Cid 0578.0454 Teb: 7ffdc000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
89de2e50 NotificationEvent
879765d0 NotificationTimer
Not impersonating
DeviceMap e10028e8
Owning Process 879767c8 Image: drwtsn32.exe
Wait Start TickCount 18102 Ticks: 9 (0:00:00:00.140)
Context Switch Count 1163
UserTime 00:00:00.203
KernelTime 00:00:00.031
Win32 Start Address drwtsn32!DispatchDebugEventThread (0x01003d6d)
Start Address kernel32!BaseThreadStartThunk (0x7c8217ec)
Stack Init f4e26000 Current f4e25be8 Base f4e26000 Limit f4e23000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr
f4e25c00 80833465 nt!KiSwapContext+0x26
f4e25c2c 80829a62 nt!KiSwapThread+0x2e5
f4e25c74 809a06ab nt!KeWaitForSingleObject+0x346
f4e25d4c 8088978c nt!NtWaitForDebugEvent+0xd5
f4e25d4c 7c9485ec nt!KiFastCallEntry+0xfc
0095ed20 60846f8f ntdll!KiFastSystemCallRet
0095ee6c 60816ecf dbgeng!LiveUserTargetInfo::WaitForEvent+0x1fa
0095ee88 608170d3 dbgeng!WaitForAnyTarget+0x45
0095eecc 60817270 dbgeng!RawWaitForEvent+0x15f
0095eee4 01003f8d dbgeng!DebugClient::WaitForEvent+0×80
0095ffb8 7c824829 drwtsn32!DispatchDebugEventThread+0×220
0095ffec 00000000 kernel32!BaseThreadStart+0×34
Knowing that a debugger suspends threads in a debuggee (Suspended Thread pattern) we see the problem process indeed:
5: kd> !process 0 2
**** NT ACTIVE PROCESS DUMP ****
...
...
...
PROCESS 898285b0 SessionId: 0 Cid: 0ca8 Peb: 7ffda000 ParentCid: 022c
DirBase: 54d5d500 ObjectTable: e2776880 HandleCount: 2.
Image: svchost.exe
THREAD 888b8668 Cid 0ca8.1448 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
SuspendCount 2
888b87f8 Semaphore Limit 0×2
Dumping its thread stacks shows only one system thread where we normally expect plenty of them originated from user space. There is also the presence of a debug port:
5: kd> .process /r /p 898285b0
Implicit process is now 898285b0
Loading User Symbols
5: kd> !process 898285b0
PROCESS 898285b0 SessionId: 0 Cid: 0ca8 Peb: 7ffda000 ParentCid: 022c
DirBase: 54d5d500 ObjectTable: e2776880 HandleCount: 2.
Image: svchost.exe
VadRoot 88953220 Vads 209 Clone 0 Private 901. Modified 3. Locked 0.
DeviceMap e10028e8
Token e27395b8
ElapsedTime 00:03:25.640
UserTime 00:00:00.156
KernelTime 00:00:00.234
QuotaPoolUsage[PagedPool] 82988
QuotaPoolUsage[NonPagedPool] 8824
Working Set Sizes (now,min,max) (2745, 50, 345) (10980KB, 200KB, 1380KB)
PeakWorkingSetSize 2819
VirtualSize 82 Mb
PeakVirtualSize 83 Mb
PageFaultCount 4519
MemoryPriority BACKGROUND
BasePriority 6
CommitCharge 1380
DebugPort 89de2e50
THREAD 888b8668 Cid 0ca8.1448 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
SuspendCount 2
888b87f8 Semaphore Limit 0x2
Not impersonating
DeviceMap e10028e8
Owning Process 898285b0 Image: svchost.exe
Wait Start TickCount 13456 Ticks: 4655 (0:00:01:12.734)
Context Switch Count 408
UserTime 00:00:00.000
KernelTime 00:00:00.000
Start Address driverA!DriverThread (0xf6fb8218)
Stack Init f455b000 Current f455a3ac Base f455b000 Limit f4558000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr
f455a3c4 80833465 nt!KiSwapContext+0x26
f455a3f0 80829a62 nt!KiSwapThread+0x2e5
f455a438 80833178 nt!KeWaitForSingleObject+0x346
f455a450 8082e01f nt!KiSuspendThread+0x18
f455a498 80833480 nt!KiDeliverApc+0x117
f455a4d0 80829a62 nt!KiSwapThread+0x300
f455a518 f6fb7f13 nt!KeWaitForSingleObject+0x346
f455a548 f4edd457 driverA!WaitForSingleObject+0x75
f455a55c f4edcdd8 driverB!DeviceWaitForRead+0x19
f455ad90 f6fb8265 driverB!InputThread+0x17e
f455adac 80949b7c driverA!DriverThread+0x4d
f455addc 8088e062 nt!PspSystemThreadStartup+0x2e
00000000 00000000 nt!KiThreadStartup+0x16
The most likely scenario was that svchost.exe experienced an unhandled exception that triggered the launch of a postmortem debugger such as Dr. Watson.
Other similar examples this pattern might include the presence of WerFault.exe on Vista, NTSD and other JIT debuggers running.
- Dmitry Vostokov @ DumpAnalysis.org -
November 24th, 2009 at 12:31 am
[…] to it exist or there are any missing threads or components inside it, any suspended threads and special processes like a postmortem debugger. We shouldn’t also forget about service dependencies and […]
June 14th, 2010 at 7:50 pm
Some processes briefly appear to do a specialized task and normally don’t present in memory. Their presence in memory dump can point to abnormal conditions such as wait chains, blocked threads, etc. For example, multiple LogonUI processes in terminal session environments on W2K8.
July 18th, 2010 at 2:08 pm
[…] sessions couldn’t be created. A complete memory dump stack trace collection log lists a special process that would not normally be present in a fully initialized session: userinit.exe. One of its […]
July 30th, 2010 at 11:09 am
Another error reporting example is the presence of DW20.exe
October 13th, 2010 at 8:03 pm
[…] Experts Magazine Online Today we introduce an icon for Special Process […]