Manual dump, wait chain, blocked thread, dynamic memory corruption and historical information: pattern cooperation
Saturday, March 14th, 2009For one manual user dump of a hanging service neither !analyze -v nor !analyze -v -hang command found the problem:
Loading Dump File [ServiceA.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER'
0:000> !analyze -v -hang
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 678.67c File IO
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 0000067c
DEFAULT_BUCKET_ID: APPLICATION_HANG_BlockedOn_FileIO
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_BlockedOn_FileIO
LAST_CONTROL_TRANSFER: from 7c82776b to 7c8285ec
STACK_TEXT:
0012fbc8 7c82776b ntdll!KiFastSystemCallRet
0012fbcc 77e218b2 ntdll!NtReadFile+0xc
0012fc34 77d85edb kernel32!ReadFile+0x16c
0012fc60 77d85f82 advapi32!ScGetPipeInput+0x2a
0012fcd4 77d71ed9 advapi32!ScDispatcherLoop+0x51
0012ff38 004025e0 advapi32!StartServiceCtrlDispatcherW+0xe3
0012ff60 0042c8fd ServiceA!main+0x110
0012ffc0 77e4f23b ServiceA!mainCRTStartup+0x143
0012fff0 00000000 kernel32!BaseProcessStart+0x23
As usual the critical section list was checked for any locked sections and to find wait chains:
0:000> !cs -l -o -s
-----------------------------------------
DebugInfo = 0x7c8877c0
Critical section = 0×7c8877a0 (ntdll!LdrpLoaderLock+0×0)
LOCKED
LockCount = 0xA2
WaiterWoken = No
OwningThread = 0×00011f64
RecursionCount = 0×1
LockSemaphore = 0×188
SpinCount = 0×00000000
OwningThread DbgId = ~20s
OwningThread Stack =
ChildEBP RetAddr Args to Child
0327f91c 7c827d0b 7c83d236 00000754 00000000 ntdll!KiFastSystemCallRet
0327f920 7c83d236 00000754 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0327f95c 7c83d281 00000754 00000004 00370000 ntdll!RtlpWaitOnCriticalSection+0×1a3
0327f97c 7c82a264 00370608 00000000 0000008c ntdll!RtlEnterCriticalSection+0xa8
0327fba4 7c3423aa 00370000 00000008 0000008c ntdll!RtlAllocateHeap+0×313
0327fbe4 7c3422cb 00000001 0000008c 00000000 msvcr71!calloc+0xe6
0327fbfc 7c81a352 7c340000 00000002 00000000 msvcr71!_CRTDLL_INIT+0×138
0327fc1c 7c82ed97 7c34229f 7c340000 00000002 ntdll!LdrpCallInitRoutine+0×14
0327fcb8 7c82ec9f 0327fd28 0327fd28 00000000 ntdll!LdrpInitializeThread+0×10d
0327fd14 7c8284c5 0327fd28 7c800000 00000000 ntdll!_LdrpInitialize+0×16f
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0×25
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
[…]
DebugInfo = 0×7c887e20
Critical section = 0×00370608 (+0×370608)
LOCKED
LockCount = 0×7
WaiterWoken = No
OwningThread = 0×000027ac
RecursionCount = 0×1
LockSemaphore = 0×754
SpinCount = 0×00000fa0
OwningThread DbgId = ~17s
OwningThread Stack =
ChildEBP RetAddr Args to Child
0170ebe0 7c82775b 77e62610 d0000144 00000004 ntdll!KiFastSystemCallRet
0170ebe4 77e62610 d0000144 00000004 00000000 ntdll!NtRaiseHardError+0xc
0170ee50 77e592a3 0170ee78 77e41ac1 0170ee80 kernel32!UnhandledExceptionFilter+0×51a
0170ee58 77e41ac1 0170ee80 00000000 0170ee80 kernel32!BaseThreadStart+0×4a
0170ee80 7c828752 0170ef64 0170ffdc 0170ef80 kernel32!_except_handler3+0×61
0170eea4 7c828723 0170ef64 0170ffdc 0170ef80 ntdll!ExecuteHandler2+0×26
0170ef4c 7c82855e 0170d000 0170ef80 0170ef64 ntdll!ExecuteHandler+0×24
0170ef4c 7c82a754 0170d000 0170ef80 0170ef64 ntdll!KiUserExceptionDispatcher+0xe (CONTEXT @ 0170ef80)
0170f258 7c82a82b 0185d6a8 00690043 0170f320 ntdll!RtlpCoalesceFreeBlocks+0×36e
0170f340 7c34218a 00370000 00000000 0185bad8 ntdll!RtlFreeHeap+0×38e
0170f388 004288df 0185bad8 05afd9c4 00000000 msvcr71!free+0xc3
0170f3b4 00428a16 05afd9c4 00001bc4 0185bad8 ServiceA!SetData+0×4df
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
[…]
We see that 162 threads (0xA2) are waiting for 0×7c8877a0 locked critical section whose owner thread #20 is waiting for the critical section 00370608 owned by the thread #17. The latter thread is blocked in an unhandled exception filter raising a hard error message box. Applying the new exception context we confirm heap corruption problem:
0:000> .cxr 0170ef80
eax=0185d6a8 ebx=00370000 ecx=00690043 edx=00000001 esi=0185d6a0 edi=01858030
eip=7c82a754 esp=0170f24c ebp=0170f258 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
ntdll!RtlpCoalesceFreeBlocks+0×36e:
7c82a754 8b09 mov ecx,dword ptr [ecx] ds:0023:00690043=????????
0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0170f258 7c82a82b ntdll!RtlpCoalesceFreeBlocks+0×36e
0170f340 7c34218a ntdll!RtlFreeHeap+0×38e
0170f388 004288df msvcr71!free+0xc3
0170f3b4 00428a16 ServiceA!SetData+0×4df
[…]
0170ffec 00000000 kernel32!BaseThreadStart+0×34
Therefore we suggest to enable full page heap either by using gflags or application verifier. Looking further at other locked critical sections omitted above we notice that one is inaccessible and from an unloaded module:
Cannot read structure field value at 0x5531953c, error 0
Cannot determine if the critical section is locked or not.
-----------------------------------------
Critical section = 0x55319538 (<Unloaded_DetouredA.DLL>+0×29538)
DebugInfo = 0×00147108
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
We see this execution residue in the list of unloaded modules too:
0:000> lmt
[...]
Unloaded modules:
[...]
697b0000 697c7000 faultrep.dll
Timestamp: Sat Feb 17 15:12:57 2007 (45D71B79)
Checksum: 0001D340
[...]
552f0000 55320000 DetouredA.DLL
Timestamp: Wed Dec 19 09:38:46 2007 (4768E6A6)
Checksum: 00030564
Recalling that this hooksware had problems before we also suggest to remove software package that injects DetouredA module to test the stability of the system.
- Dmitry Vostokov @ DumpAnalysis.org -
