Early crash dump, blocked thread, not my version and lost opportunity: pattern cooperation
It was reported that one important Windows service stops responding from time to time. The customer was proactive in gathering memory dumps and we got several early crash dumps. Most of them were false positive showing normal error handling via throwing an exception:
0:042> kL
ChildEBP RetAddr
0f7bec6c 77c31e37 kernel32!RaiseException+0×53
0f7bec84 77c32042 rpcrt4!RpcpRaiseException+0×24
0f7bec94 77cb30e4 rpcrt4!NdrGetBuffer+0×46
0f7bf080 09a554a6 rpcrt4!NdrClientCall2+0×197
[…]
However one such dump also had a clearly blocked thread which was blocking 10 different threads:
0:042> !locks
CritSec MyService!MainCriticalSection+0 at 0041b9a0
WaiterWoken No
LockCount 0
RecursionCount 1
OwningThread ad0
EntryCount 0
ContentionCount 0
*** Locked
CritSec +339fb8 at 00339fb8
WaiterWoken No
LockCount 10
RecursionCount 1
OwningThread ad0
EntryCount 0
ContentionCount 31
*** Locked
0:042> ~~[ad0]kL
ChildEBP RetAddr
008dc1e0 7c94734b ntdll!KiFastSystemCallRet
008dc1e4 77d96c61 ntdll!NtOpenKey+0xc
008dc244 77d8e15f advapi32!LocalBaseRegOpenKey+0xd0
008dc278 6064fe47 advapi32!RegOpenKeyExA+0×11c
WARNING: Stack unwind information not available. Following frames may be wrong.
008dc8b8 6064fa00 NotMyDLL!getvar+0×4e7
[…]
Checking NotMyDLL module time stamp we identified Not My Version pattern because we expected much later version:
0:042> lmt m NotMyDLL
start end module name
60600000 60686000 NotMyDLL Mon Oct 30 10:14:07 1999
We know this component often had problems in the past and although being stuck in registry access could be a coincidence, registry corruption or system-wide problem we immediately advised to upgrade the component to the latest stable version. We also got a manual dump of the service when the customer tried to restart it and it showed the signs of Lost Opportunity pattern:
0:000> kv
ChildEBP RetAddr Args to Child
1744fd44 7c947d0b 7c821d1e 00001b58 00000000 ntdll!KiFastSystemCallRet
1744fd48 7c821d1e 00001b58 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
1744fdb8 7c821c8d 00001b58 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
1744fdcc 67e223dd 00001b58 ffffffff 1744fdf4 kernel32!WaitForSingleObject+0x12
1744fde0 7c93a352 67e20000 00000000 00000001 MyDLL!DllInitialize+0xed
1744fe00 7c950e70 67e222f0 67e20000 00000000 ntdll!LdrpCallInitRoutine+0x14
1744feb8 7c8268a3 00000000 00000000 00000000 ntdll!LdrShutdownProcess+0x182
1744ffa4 7c826905 c0000005 77e8f3b0 ffffffff kernel32!_ExitProcess+0x43
1744ffb8 7c8392c1 c0000005 00000000 00000000 kernel32!ExitProcess+0×14
1744ffec 00000000 77c4b0f5 0b644720 00000000 kernel32!BaseThreadStart+0×5f
0:000> !teb
TEB at 7ff4b000
ExceptionList: 1744fda8
StackBase: 17450000
StackLimit: 17449000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ff4b000
EnvironmentPointer: 00000000
ClientId: 00001e90 . 00001168
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdd000
LastErrorValue: 0
LastStatusValue: 103
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 17449000 17450000
[...]
1744f4b0 7c94775b ntdll!NtRaiseHardError+0xc
1744f4b4 7c842610 kernel32!UnhandledExceptionFilter+0×51a
1744f4b8 d0000144
1744f4bc 00000000
[…]
0:000> !error d0000144
Error code: (NTSTATUS) 0xd0000144 (3489661252) - {Application Error} The exception %s (0x%08lx) occurred in the application at location 0x%08lx.
Therefore we additionally advised to dump the process manually using userdump.exe when an error message box appears on the console session. We hope that getting right dump files at the right time via the right method would prove or disprove our hypothesis about NotMyDLL component.
- Dmitry Vostokov @ DumpAnalysis.org -