Early crash dump, blocked thread, not my version and lost opportunity: pattern cooperation

It was reported that one important Windows service stops responding from time to time. The customer was proactive in gathering memory dumps and we got several early crash dumps. Most of them were false positive showing normal error handling via throwing an exception:

0:042> kL
ChildEBP RetAddr 
0f7bec6c 77c31e37 kernel32!RaiseException+0×53
0f7bec84 77c32042 rpcrt4!RpcpRaiseException+0×24
0f7bec94 77cb30e4 rpcrt4!NdrGetBuffer+0×46
0f7bf080 09a554a6 rpcrt4!NdrClientCall2+0×197
[…]

However one such dump also had a clearly blocked thread which was blocking 10 different threads:

0:042> !locks

CritSec MyService!MainCriticalSection+0 at 0041b9a0
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       ad0
EntryCount         0
ContentionCount    0
*** Locked

CritSec +339fb8 at 00339fb8
WaiterWoken        No
LockCount          10
RecursionCount     1
OwningThread       ad0
EntryCount         0
ContentionCount    31
*** Locked

0:042> ~~[ad0]kL
ChildEBP RetAddr 
008dc1e0 7c94734b ntdll!KiFastSystemCallRet
008dc1e4 77d96c61 ntdll!NtOpenKey+0xc
008dc244 77d8e15f advapi32!LocalBaseRegOpenKey+0xd0
008dc278 6064fe47 advapi32!RegOpenKeyExA+0×11c
WARNING: Stack unwind information not available. Following frames may be wrong.
008dc8b8 6064fa00 NotMyDLL!getvar+0×4e7
[…]

Checking NotMyDLL module time stamp we identified Not My Version pattern because we expected much later version:

0:042> lmt m NotMyDLL
start    end        module name
60600000 60686000   NotMyDLL  Mon Oct 30 10:14:07 1999

We know this component often had problems in the past and although being stuck in registry access could be a coincidence, registry corruption or system-wide problem we immediately advised to upgrade the component to the latest stable version. We also got a manual dump of the service when the customer tried to restart it and it showed the signs of Lost Opportunity pattern:

0:000> kv
ChildEBP RetAddr Args to Child
1744fd44 7c947d0b 7c821d1e 00001b58 00000000 ntdll!KiFastSystemCallRet
1744fd48 7c821d1e 00001b58 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
1744fdb8 7c821c8d 00001b58 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
1744fdcc 67e223dd 00001b58 ffffffff 1744fdf4 kernel32!WaitForSingleObject+0x12
1744fde0 7c93a352 67e20000 00000000 00000001 MyDLL!DllInitialize+0xed
1744fe00 7c950e70 67e222f0 67e20000 00000000 ntdll!LdrpCallInitRoutine+0x14
1744feb8 7c8268a3 00000000 00000000 00000000 ntdll!LdrShutdownProcess+0x182
1744ffa4 7c826905 c0000005 77e8f3b0 ffffffff kernel32!_ExitProcess+0x43
1744ffb8 7c8392c1 c0000005 00000000 00000000 kernel32!ExitProcess+0×14
1744ffec 00000000 77c4b0f5 0b644720 00000000 kernel32!BaseThreadStart+0×5f

0:000> !teb
TEB at 7ff4b000
    ExceptionList:        1744fda8
    StackBase:            17450000
    StackLimit:           17449000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ff4b000
    EnvironmentPointer:   00000000
    ClientId:             00001e90 . 00001168
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffdd000
    LastErrorValue:       0
    LastStatusValue:      103
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 17449000 17450000
[...]
1744f4b0  7c94775b ntdll!NtRaiseHardError+0xc
1744f4b4  7c842610 kernel32!UnhandledExceptionFilter+0×51a
1744f4b8  d0000144
1744f4bc  00000000
[…]

0:000> !error d0000144
Error code: (NTSTATUS) 0xd0000144 (3489661252) - {Application Error} The exception %s (0x%08lx) occurred in the application at location 0x%08lx.

Therefore we additionally advised to dump the process manually using userdump.exe when an error message box appears on the console session. We hope that getting right dump files at the right time via the right method would prove or disprove our hypothesis about NotMyDLL component.

- Dmitry Vostokov @ DumpAnalysis.org -

Leave a Reply