Crash Dump Analysis Patterns (Part 72)
It is important to save crash dumps at the right time, for example, when an error message box is shown as discussed in one of my previous posts:
Crash Dumps for Dummies (Part 7)
However, sometimes crash dumps are saved after a visual indicator of the problem disappeared and the opportunity to see the stack trace was gone. This pattern is called Lost Opportunity. Here is one example of a service memory dump saved manually after it became unresponsive. In the dump file there was only one thread left excluding the thread created by a debugger:
0:001> ~*kv
0 Id: a3c.700 Suspend: 1 Teb: 7ff59000 Unfrozen
ChildEBP RetAddr Args to Child
1178fd60 7c822124 77e6bad8 00000574 00000000 ntdll!KiFastSystemCallRet
1178fd64 77e6bad8 00000574 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
1178fdd4 77e6ba42 00000574 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
1178fde8 67e223dd 00000574 ffffffff 1178fe10 kernel32!WaitForSingleObject+0x12
1178fdfc 7c82257a 67e20000 00000000 00000001 componentA!DllInitialize+0xed
1178fe1c 7c8118b0 67e222f0 67e20000 00000000 ntdll!LdrpCallInitRoutine+0x14
1178feb8 77e53002 00000000 00000000 00000000 ntdll!LdrShutdownProcess+0x130
1178ffa4 77e53065 c0000005 77e8f3b0 ffffffff kernel32!_ExitProcess+0×43
1178ffb8 77e84277 c0000005 00000000 00000000 kernel32!ExitProcess+0×14
1178ffec 00000000 77c5de6d 0a078138 00000000 kernel32!BaseThreadStart+0×5f
# 1 Id: a3c.18bc Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr Args to Child
0a6cffc8 7c845ea0 00000005 00000004 00000001 ntdll!DbgBreakPoint
0a6cfff4 00000000 00000000 00905a4d 00000003 ntdll!DbgUiRemoteBreakin+0x36
We also see exception code 0xc0000005 as ExitProcess parameter. The raw stack reveals the call to NtRaiseHardError function that definitely resulted in some error message box:
0:001> ~0s
[...]
0:000> !teb
TEB at 7ff59000
ExceptionList: 1178fdc4
StackBase: 11790000
StackLimit: 11789000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ff59000
EnvironmentPointer: 00000000
ClientId: 00000a3c . 00000700
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: c0000008
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 11789000 11790000
11789000 00000000
11789004 00000000
11789008 00000000
1178900c 00000000
11789010 00000000
[...]
1178f058 0a4016f4
1178f05c 1178efe0
1178f060 695040c4 <Unloaded_faultrep.dll>+0x40c4
1178f064 7ffdf000
1178f068 00000000
1178f06c 0a4016f4
1178f070 0a4016f4
1178f074 00000000
1178f078 1178f06c
1178f07c 0a4016b8
1178f080 0a4016b8
1178f084 1178efa0
1178f088 7c821b74 ntdll!NtRaiseHardError+0xc
1178f08c 77e99af9 kernel32!UnhandledExceptionFilter+0×54b
1178f090 d0000144
1178f094 00000004
1178f098 00000000
1178f09c 1178f164
1178f0a0 00000001
1178f0a4 77e996a7 kernel32!UnhandledExceptionFilter+0×873
1178f0a8 00000000
1178f0ac 00000000
1178f0b0 00000000
1178f0b4 02f049f0
1178f0b8 1178f13c
1178f0bc 00000000
[…]
It was that time when the dump should have been saved. See also Process crash - getting the dump manually post for another example and full explanation.
- Dmitry Vostokov @ DumpAnalysis.org -
October 9th, 2008 at 4:46 pm
[…] got a manual dump of the service when the customer tried to restart it and it showed the signs of Lost Opportunity […]