Software Diagnostics Library » Crash Dump Patterns

Truncated dump, stack trace collection, waiting thread time and wait chains: pattern cooperation

Thursday, September 10th, 2009

It was reported that functionality of some applications was not available until ServiceA was killed and restarted. Complete memory dump was forced. For that memory dump WinDbg !stacks command shows every thread bugchecking the system, for example:

0: kd> !stacks Proc.Thread .Thread Ticks ThreadState Blocker [...] [8ca72020 ApplicationA.exe] 4bc.0004c4 8bf76af0 0000cc4 Blocked nt!KeBugCheckEx+0x1b 4bc.0004d0 8bf53650 000055d Blocked nt!KeBugCheckEx+0x1b 4bc.0004e8 8bf33c88 0000094 Blocked nt!KeBugCheckEx+0x1b 4bc.001ffc 8ba86020 0000eed Blocked nt!KeBugCheckEx+0x1b 4bc.001e30 8b7a0ca8 00003ac Blocked nt!KeBugCheckEx+0x1b 4bc.001934 8b664020 0000eed Blocked nt!KeBugCheckEx+0x1b 4bc.001af0 8ca6d3b0 0000094 Blocked nt!KeBugCheckEx+0x1b 4bc.001cac 8bf41af0 0001412 Blocked nt!KeBugCheckEx+0x1b 4bc.00141c 8b6458d0 0000094 Blocked nt!KeBugCheckEx+0x1b [...]

Seeing this nonsense I checked that complete dump was truncated by half because page file was 4Gb but the amount of physical memory was 8Gb:

0: kd> !vm *** Virtual Memory Usage *** Physical Memory: 2096553 ( 8386212 Kb) Paging File Name paged out Current: 4195328 Kb Free Space: 4182336 Kb Minimum: 4195328 Kb Maximum: 4195328 Kb […]

However it was possible to get stack trace collection using !process 0 ff command with most data from _ETHREAD structures present but most of kernel and user space stack traces not present in the output. ServiceA has many threads waiting for LPC requests during last 5 minutes (the bugcheck was forced after 3 - 4 minutes the issue manifested itself):

THREAD 8cc9b590 Cid 053c.0668 Teb: 7ffaf000 Win32Thread: b84e5770 WAIT: (Unknown) UserMode Non-Alertable 8cc9b77c Semaphore Limit 0x1 Waiting for reply to LPC MessageId 0005bbdf: Current LPC port e2cee338 IRP List: 8b6548b8: (0006,0268) Flags: 00000000 Mdl: 00000000 Not impersonating DeviceMap e1003860 Owning Process 8bf81d88 Image: ServiceA.exe Attached Process N/A Image: N/A Wait Start TickCount 193823 Ticks: 13752 (0:00:03:34.875) Context Switch Count 389 LargeStack UserTime 00:00:00.015 KernelTime 00:00:00.000 Win32 Start Address 0×7d1f5e70 Start Address 0×77e617ec Stack Init b3a44000 Current b3a43c08 Base b3a44000 Limit b3a41000 Call 0 Priority 9 BasePriority 8 PriorityDecrement 0 Kernel stack not resident.

The similar waits were found for ApplicationB and ApplicationC processes. Therefore it was advised to limit the amount of physical memory to 4Gb and reproduce the issue or if the liveliness of the system needs to be preserved to manually dump the following processes on the next occasion:

ServiceA.exe
ApplicationB.exe
ApplicationC.exe

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns | No Comments »

Wild code and partial stack reconstruction

Friday, September 4th, 2009

I recently got a chance to see an instance of Wild Code pattern in kernel mode:

ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY (fc) An attempt was made to execute non-executable memory. The guilty driver is on the stack trace (and is typically the current instruction pointer). When possible, the guilty driver's name (Unicode string) is printed on the bugcheck screen and saved in KiBugCheckDriver. Arguments: Arg1: a98ccfc4, Virtual address for the attempted execute. Arg2: 17b1b963, PTE contents. Arg3: a98ccf38, (reserved) Arg4: 00000001, (reserved)

3: kd> .trap 0xffffffffa98ccf38 ErrCode = 00000011 eax=00000000 ebx=bf8c16eb ecx=bf855770 edx=00000026 esi=be4da0e8 edi=0000029e eip=a98ccfc4 esp=a98ccfac ebp=a98ccfa0 iopl=0 nv up ei ng nz ac pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010296 a98ccfc4 dcda fcomp3 st(2)

3: kd> !pte a98ccfc4 VA a98ccfc4 PDE at 00000000C0602A60 PTE at 00000000C054C660 contains 000000000B40A863 contains 8000000017B1B963 pfn b40a ---DA--KWEV pfn 17b1b -G-DA--KW-V

3: kd> k *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. a98ccff0 80889db4 0xa98ccfc4 a98ccff0 00000000 nt!KiCallbackReturn+0×84

We see that the execution address belongs to thread kernel stack range:

3: kd> !thread THREAD 8959f548 Cid 20c4.0dd8 Teb: 7ffdf000 Win32Thread: bc1a9550 RUNNING on processor 3 Not impersonating DeviceMap e183a628 Owning Process 890af4b0 Image: ApplicationA.exe Attached Process N/A Image: N/A Wait Start TickCount 67969974 Ticks: 0 Context Switch Count 2833569 LargeStack UserTime 00:01:18.171 KernelTime 00:00:50.468 Win32 Start Address 0x00401c94 Start Address 0x7c8217f8 Stack Init a98cd260 Current a98ccc48 Base a98ce000 Limit a98c3000 Call a98cd268 Priority 13 BasePriority 8 PriorityDecrement 0 ChildEBP RetAddr Args to Child a98ccea8 8085eced 000000fc a98ccfc4 17b1b963 nt!KeBugCheckEx+0×1b a98ccf20 8088c798 00000008 a98ccfc4 00000000 nt!MmAccessFault+0xb25 a98ccf20 a98ccfc4 00000008 a98ccfc4 00000000 nt!KiTrap0E+0xdc WARNING: Frame IP not in any known module. Following frames may be wrong. a98ccff0 80889db4 0013e3d4 0000000c 00000000 0xa98ccfc4 a98ccff0 00000000 0013e3d4 0000000c 00000000 nt!KiCallbackReturn+0×84

Something must have gone wrong after the return from KiCallbackReturn. On x86 systems this is an IDT entry (2b). See an example output I did while writing down notes on Windows Internals. Windows NT/2000 Native API Reference states that it is the same function as ZwCallBackReturn and it is used to return from win32k.sys user-mode and space callbacks, for example, to send a window message (p. 408). We find the following call on raw stack and use an extended version of k command to get partial stack trace before user-mode callback call:

3: kd> dds a98c3000 a98ce000 [...] a98cd244 a98cd270 a98cd248 80833485 nt!KiSwapThread+0x305 a98cd24c 8959f548 a98cd250 8959f5f0 a98cd254 00000001 a98cd258 8959f548 a98cd25c 00000000 a98cd260 ffffffff a98cd264 00000001 a98cd268 a98cd8a0 a98cd26c a98cd604 a98cd270 a98cd8a8 a98cd274 0013f2e4 a98cd278 7ffdf000 a98cd27c 0013e3ac a98cd280 a98cd2d8 a98cd284 8091d6d1 nt!KeUserModeCallback+0×8f a98cd288 a98cd388 a98cd28c a98cd384 a98cd290 a98cd370 a98cd294 bc1a9550 a98cd298 006fa0e8 a98cd29c 0013f2e4 a98cd2a0 7ffdf000 a98cd2a4 00000018 a98cd2a8 8948c7f8 […]

3: kd> k L=a98cd280 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. a98cd280 8091d6d1 0xa98ccfc4 a98cd2d8 bf858a9a nt!KeUserModeCallback+0×8f a98cd3b4 bf8a244e win32k!SfnINOUTNCCALCSIZE+0×10b a98cd3fc bf8a13fa win32k!xxxSendMessageToClient+0×176 a98cd448 bf8a130f win32k!xxxSendMessageTimeout+0×1a6 a98cd46c bf85cd68 win32k!xxxSendMessage+0×1b a98cd530 bf85daca win32k!xxxCalcValidRects+0×3bc a98cd58c bf85def4 win32k!xxxEndDeferWindowPosEx+0xf2 a98cd5a8 bf85061c win32k!xxxSetWindowPos+0xb1 a98cd5cc bf8e3df8 win32k!xxxRedrawFrame+0×16 a98cd5d8 bf8a2b69 win32k!xxxDrawMenuBar+0×19 a98cd5f4 8088978c win32k!NtUserCallHwndLock+0×6b a98cd5f4 7c9485ec nt!KiFastCallEntry+0xfc 0013e438 00000000 0×7c9485ec

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

10 Common Mistakes in Memory Analysis (Part 5)

Monday, August 31st, 2009

Sometimes not paying attention to all aspects of default analysis makes it difficult to consider an alternative troubleshooting hypothesis. Here is a sample of !analyze -v output showing massive patching (hooked functions pattern) by DriverA module:

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e) This is a very common bugcheck. Usually the exception address pinpoints the driver/function that caused the problem. Always note this address as well as the link date of the driver/image that contains this address. Some common problems are exception code 0x80000003. This means a hard coded breakpoint or assertion was hit, but this system was booted /NODEBUG. This is not supposed to happen as developers should never have hardcoded breakpoints in retail code, but ... If this happens, make sure a debugger gets connected, and the system is booted /DEBUG. This will let us see why this breakpoint is happening. Arguments: Arg1: c0000005, The exception code that was not handled Arg2: 8092d47f, The address that the exception occurred at Arg3: f5205b14, Trap Frame Arg4: 00000000

[...]

CHKIMG_EXTENSION: !chkimg -lo 50 -d !nt 80822a49-80822a4d 5 bytes - nt!NtYieldExecution [ 8b ff 55 8b ec:e9 14 3a 95 76 ] 80823c11-80823c14 4 bytes - nt!KeFlushProcessTb+2c (+0x11c8) [ 69 76 82 80:88 ff ff ff ] 80823c17-80823c1a 4 bytes - nt!KeFlushProcessTb+32 (+0x06) [ dd 40 01 00:b5 34 b3 76 ] 8083771f-80837725 7 bytes - nt!KeAcquireQueuedSpinLockAtDpcLevel+1b (+0x13b08) [ f7 41 04 01 00 00 00:e9 c4 f9 b1 76 cc cc ] 80840945-8084094a 6 bytes - nt!KxFlushEntireTb+9 (+0x9226) [ ff 15 1c 10 80 80:e9 65 67 b1 76 cc ] 80845fe0-80845fe3 4 bytes - nt!KeFlushSingleTb+49 (+0x569b) [ 14 1d ff ff:dd 10 b1 76 ] 80845fe5 - nt!KeFlushSingleTb+4e (+0x05) [ b9:c3 ] 8084722d-80847230 4 bytes - nt!KeFlushMultipleTb+45 (+0x1248) [ 5e e3 82 80:14 00 00 00 ] 80847233-80847236 4 bytes - nt!KeFlushMultipleTb+4b (+0x06) [ c1 0a ff ff:99 fe b0 76 ] 808c039c-808c039e 3 bytes - nt!NtSetContextThread [ 8b ff 55:e9 31 5f ] 808c03a0 - nt!NtSetContextThread+4 (+0x04) [ ec:76 ] 808e3184-808e3188 5 bytes - nt!NtCreateProcess (+0x22de4) [ 8b ff 55 8b ec:e9 0b 31 89 76 ] 808f6ad0-808f6ad6 7 bytes - nt!NtLoadKeyEx (+0x1394c) [ 6a 70 68 98 4b 81 80:e9 e7 f8 87 76 90 90 ] 8090c66f-8090c675 7 bytes - nt!NtDeleteValueKey (+0x15b9f) [ 6a 44 68 60 f0 81 80:e9 c4 9c 86 76 90 90 ] 8090e36c-8090e370 5 bytes - nt!NtTerminateProcess (+0x1cfd) [ 8b ff 55 8b ec:e9 34 81 86 76 ] 80915342-80915346 5 bytes - nt!NtDeleteKey (+0x6fd6) [ 8b ff 55 8b ec:e9 c7 0f 86 76 ] 80918114-80918118 5 bytes - nt!NtOpenThread (+0x2dd2) [ 68 c4 00 00 00:e9 53 e1 85 76 ] 80921eac-80921eb2 7 bytes - nt!NtEnumerateKey (+0x9d98) [ 6a 48 68 f0 f9 82 80:e9 f5 44 85 76 90 90 ] 80922578-8092257e 7 bytes - nt!NtEnumerateValueKey (+0x6cc) [ 6a 48 68 10 fc 82 80:e9 13 3e 85 76 90 90 ] 80922efd-80922f01 5 bytes - nt!NtNotifyChangeKey (+0x985) [ 8b ff 55 8b ec:e9 e4 34 85 76 ] 809246fb-809246ff 5 bytes - nt!NtOpenProcess (+0x17fe) [ 68 c8 00 00 00:e9 58 1b 85 76 ] 8092c8a0-8092c8a4 5 bytes - nt!NtCreateKey (+0x81a5) [ 68 c0 00 00 00:e9 55 9a 84 76 ] 8092f3a6-8092f3ac 7 bytes - nt!NtSetValueKey (+0x2b06) [ 6a 58 68 a0 f6 82 80:e9 a3 6f 84 76 90 90 ] 8092fa88-8092fa8c 5 bytes - nt!NtCreateFile (+0x6e2) [ 8b ff 55 8b ec:e9 ab 69 84 76 ] 80931311-80931315 5 bytes - nt!NtOpenKey (+0x1889) [ 68 ac 00 00 00:e9 d0 4f 84 76 ] 809316ed-809316f3 7 bytes - nt!NtQueryValueKey (+0x3dc) [ 6a 60 68 80 90 84 80:e9 72 4c 84 76 90 90 ] 8093470f-80934715 7 bytes - nt!NtQueryKey (+0x3022) [ 6a 58 68 c8 97 84 80:e9 0e 1d 84 76 90 90 ] 809354fa-80935500 7 bytes - nt!NtMapViewOfSection (+0xdeb) [ 6a 38 68 80 a2 84 80:e9 77 0f 84 76 90 90 ] 80935785-80935789 5 bytes - nt!NtUnmapViewOfSection (+0x28b) [ 8b ff 55 8b ec:e9 02 0d 84 76 ] 8093ba96-8093ba9c 7 bytes - nt!NtProtectVirtualMemory (+0x6311) [ 6a 44 68 40 03 85 80:e9 b1 a9 83 76 90 90 ] 8093c86d-8093c871 5 bytes - nt!NtSetInformationProcess (+0xdd7) [ 68 08 01 00 00:e9 4c 9a 83 76 ] 8093ce6b-8093ce71 7 bytes - nt!NtCreateProcessEx (+0x5fe) [ 6a 0c 68 58 0e 85 80:e9 38 94 83 76 90 90 ] 80978fef-80978ff5 7 bytes - nt!NtQueryMultipleValueKey (+0x3c184) [ 6a 48 68 f0 f9 86 80:e9 86 d3 7f 76 90 90 ] 80979775-8097977b 7 bytes - nt!NtRenameKey (+0x786) [ 6a 3c 68 38 fa 86 80:e9 a8 cb 7f 76 90 90 ] 80979caf-80979cb3 5 bytes - nt!NtRestoreKey (+0x53a) [ 8b ff 55 8b ec:e9 46 c7 7f 76 ] 8097a11c-8097a120 5 bytes - nt!NtUnloadKey (+0x46d) [ 8b ff 55 8b ec:e9 b1 c2 7f 76 ] 8097a139-8097a13d 5 bytes - nt!NtReplaceKey (+0x1d) [ 8b ff 55 8b ec:e9 d0 c2 7f 76 ] 197 errors : !nt (80822a49-8097a13d)

MODULE_NAME: DriverA

IMAGE_NAME: DriverA.sys

MEMORY_CORRUPTOR: PATCH_DriverA

FAILURE_BUCKET_ID: MEMORY_CORRUPTION_PATCH_DriverA

BUCKET_ID: MEMORY_CORRUPTION_PATCH_DriverA

However, when we look at the stack trace, we would see that BSOD happened when accessing registry while updating drivers:

FAULTING_IP: nt!HvpGetCellMapped+97 8092d47f 8b4604 mov eax,dword ptr [esi+4]

TRAP_FRAME: f5205b14 -- (.trap 0xfffffffff5205b14) ErrCode = 00000000 eax=e1021000 ebx=e101a3b8 ecx=00000003 edx=89214988 esi=00000100 edi=00000000 eip=8092d47f esp=f5205b88 ebp=f5205bfc iopl=0 nv up ei pl nz na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010206 nt!HvpGetCellMapped+0×97: 8092d47f 8b4604 mov eax,dword ptr [esi+4] ds:0023:00000104=???????? Resetting default scope

PROCESS_NAME: updatedrivers.exe

STACK_TEXT: f52056e0 8085bb9f 0000008e c0000005 8092d47f nt!KeBugCheckEx+0x1b f5205aa4 808346b4 f5205ac0 00000000 f5205b14 nt!KiDispatchException+0x3a2 f5205b0c 80834668 f5205bfc 8092d47f badb0d00 nt!CommonDispatchException+0x4a f5205b98 8092d559 e101a3b8 e63a8e40 0010fc18 nt!Kei386EoiHelper+0x186 f5205bfc 80920fcd e101a3b8 00610052 3b9aca07 nt!HvpGetCellMapped+0×36a f5205c20 8092248b e63a8e40 e22b4794 00000000 nt!CmpGetValueKeyFromCache+0xa4 f5205cc0 80922649 e63a8e40 00000000 00000001 nt!CmEnumerateValueKey+0×45a f5205d44 80833bdf 00000058 00000000 00000001 nt!NtEnumerateValueKey+0×1c9 f5205d44 7c9485ec 00000058 00000000 00000001 nt!KiFastCallEntry+0xfc WARNING: Frame IP not in any known module. Following frames may be wrong. 001290fc 00000000 00000000 00000000 00000000 0×7c9485ec

So an alternative hypothesis to pursue would be some sort of registry corruption after driver updates.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Common Mistakes, Crash Dump Analysis, Crash Dump Patterns | No Comments »

Stack trace collection, suspended threads, not my version, special process, main thread and blocked LPC chain threads: pattern cooperation

Tuesday, August 11th, 2009

It was reported that one server was hanging during automated reboot. Stack trace collection shows a few suspended and frozen threads. They all belong to the same process, ServiceA:

PROCESS 8545eb18 SessionId: 0 Cid: 0fec Peb: 7ffd4000 ParentCid: 0fdc DirBase: 3fbeb8e0 ObjectTable: e19dd1d0 HandleCount: 169. Image: ServiceA.exe

THREAD 859cc900 Cid 0fec.0ff0 Teb: 7ffdf000 Win32Thread: bc1738d0 WAIT: (Unknown) KernelMode Non-Alertable SuspendCount 1 FreezeCount 1 859cca90 Semaphore Limit 0×2

THREAD 858c6480 Cid 0fec.0ff4 Teb: 7ffde000 Win32Thread: bc178c40 WAIT: (Unknown) KernelMode Non-Alertable SuspendCount 1 f55747d8 SynchronizationEvent

THREAD 859f2338 Cid 0fec.0ff8 Teb: 7ffdd000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable SuspendCount 1 FreezeCount 1 859f24c8 Semaphore Limit 0×2

THREAD 859be2b8 Cid 0fec.0ffc Teb: 7ffdc000 Win32Thread: bc1915d8 WAIT: (Unknown) KernelMode Non-Alertable SuspendCount 1 FreezeCount 1 859be448 Semaphore Limit 0×2

[...]

When zooming into this process we see that one thread was processing an exception:

0: kd> .process /r /p 8545eb18 Implicit process is now 8545eb18 Loading User Symbols

0: kd> !process 8545eb18 [...] THREAD 858c6480 Cid 0fec.0ff4 Teb: 7ffde000 Win32Thread: bc178c40 WAIT: (Unknown) KernelMode Non-Alertable SuspendCount 1 f55747d8 SynchronizationEvent Not impersonating DeviceMap e10008e8 Owning Process 8545eb18 Image: ServiceA.exe Attached Process N/A Image: N/A Wait Start TickCount 6927 Ticks: 89866 (0:00:23:24.156) Context Switch Count 156 LargeStack UserTime 00:00:00.031 KernelTime 00:00:00.000 Win32 Start Address 0x611054cb Start Address kernel32!BaseThreadStartThunk (0x7c8217ec) Stack Init f5575000 Current f557471c Base f5575000 Limit f5571000 Call 0 Priority 10 BasePriority 8 PriorityDecrement 0 ChildEBP RetAddr f5574734 80833ec5 nt!KiSwapContext+0x26 f5574760 80829c14 nt!KiSwapThread+0x2e5 f55747a8 809a25c8 nt!KeWaitForSingleObject+0x346 f5574888 809a3739 nt!DbgkpQueueMessage+0x178 f55748ac 809a386e nt!DbgkpSendApiMessage+0x45 f5574938 8082d7ec nt!DbgkForwardException+0x90 f5574cf4 8088bed2 nt!KiDispatchException+0×1ea f5574d5c 8088be86 nt!CommonDispatchException+0×4a f5574da0 7c829c3a nt!Kei386EoiHelper+0×186 f5574dd0 00000000 kernel32!LoadResource+0×5d

We zoom into its parameters in search of semantically consistent output of .exr, .cxr and .trap commands:

0: kd> .thread 858c6480 Implicit thread is now 858c6480

0: kd> kv 100 ChildEBP RetAddr Args to Child f5574734 80833ec5 858c6480 858c6528 00000200 nt!KiSwapContext+0x26 f5574760 80829c14 00000000 858c6480 f55747d0 nt!KiSwapThread+0x2e5 f55747a8 809a25c8 f55747d8 00000000 00000000 nt!KeWaitForSingleObject+0x346 f5574888 809a3739 8545eb18 00000000 f55748c0 nt!DbgkpQueueMessage+0x178 f55748ac 809a386e f55748c0 00000001 f5574d64 nt!DbgkpSendApiMessage+0x45 f5574938 8082d7ec f5574d10 00000001 00000000 nt!DbgkForwardException+0x90 f5574cf4 8088bed2 f5574d10 00000000 f5574d64nt!KiDispatchException+0×1ea f5574d5c 8088be86 005bf4b4 61213267 badb0d00 nt!CommonDispatchException+0×4a f5574da0 7c829c3a 71c22898 00000001 ffffffff nt!Kei386EoiHelper+0×186 f5574dd0 00000000 005bf448 00000023 00000000 kernel32!LoadResource+0×5d

After probing parameters for KiDispatchException we get these results pointing to ModuleA:

0: kd> .exr f5574d10 ExceptionAddress: 61213267 (ModuleA!GetData+0×0000b57f) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: 71c22898 Attempt to read from address 71c22898

0: kd> .trap f5574d64 ErrCode = 00000004 eax=71c22898 ebx=0073a7a8 ecx=7c829c3a edx=71c1c000 esi=00000104 edi=00000000 eip=61213267 esp=005bf448 ebp=005bf4b4 iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 ModuleA!GetData+0×0000b57f: 001b:61213267 0fb700 movzx eax,word ptr [eax] ds:0023:71c22898=????

We check its data using lmv WinDbg command and find out that it is old and needs to be updated. But we don’t stop our investigation here. The fact that ServiceA was suspended means that it was probably being debugged or memory dumped. And indeed, we see NTSD in a process list:

0: kd> !process 0 0 **** NT ACTIVE PROCESS DUMP **** PROCESS 8619d5d0 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000 DirBase: 3fbeb020 ObjectTable: e1001e08 HandleCount: 1651. Image: System

PROCESS 85e95d88 SessionId: none Cid: 019c Peb: 7ffdf000 ParentCid: 0004 DirBase: 3fbeb040 ObjectTable: e16d5f18 HandleCount: 19. Image: smss.exe

PROCESS 85e4fd88 SessionId: 0 Cid: 01cc Peb: 7ffd4000 ParentCid: 019c DirBase: 3fbeb060 ObjectTable: e1561d70 HandleCount: 907. Image: csrss.exe

PROCESS 85e42d88 SessionId: 0 Cid: 01e4 Peb: 7ffde000 ParentCid: 019c DirBase: 3fbeb080 ObjectTable: e16a97b0 HandleCount: 504. Image: winlogon.exe

[...]

PROCESS 85a4dd18 SessionId: 0 Cid: 0fdc Peb: 7ffda000 ParentCid: 0214 DirBase: 3fbeb520 ObjectTable: e1aa5b38 HandleCount: 121. Image: ntsd.exe

[...]

If we zoom into NTSD process we would see that its main thread was waiting for a console input:

0: kd> !process 0fdc ff [...] THREAD 859f8768 Cid 0fdc.0fe0 Teb: 7ffdf000 Win32Thread: bc14cb38 WAIT: (Unknown) UserMode Non-Alertable 859f8954 Semaphore Limit 0x1 Waiting for reply to LPC MessageId 00001f98: Current LPC port e19f03a0 Not impersonating DeviceMap e10008e8 Owning Process 85a4dd18 Image: ntsd.exe Attached Process N/A Image: N/A Wait Start TickCount 6932 Ticks: 89861 (0:00:23:24.078) Context Switch Count 450 LargeStack UserTime 00:00:00.000 KernelTime 00:00:00.078 Win32 Start Address ntsd!mainCRTStartup (0×0100845a) Start Address kernel32!BaseProcessStartThunk (0×7c8217f8) Stack Init f55c5000 Current f55c4c08 Base f55c5000 Limit f55c1000 Call 0 Priority 13 BasePriority 13 PriorityDecrement 0 Kernel stack not resident. ChildEBP RetAddr f55c4c20 80833ec5 nt!KiSwapContext+0×26 f55c4c4c 80829c14 nt!KiSwapThread+0×2e5 f55c4c94 80920fba nt!KeWaitForSingleObject+0×346 f55c4d50 8088b3fc nt!NtRequestWaitReplyPort+0×776 f55c4d50 7c94860c nt!KiFastCallEntry+0xfc 0006ece0 7c947899 ntdll!KiFastSystemCallRet 0006ece4 7c94ec4a ntdll!ZwRequestWaitReplyPort+0xc 0006ed04 7c80cf8c ntdll!CsrClientCallServer+0×8c 0006edfc 7c872904 kernel32!ReadConsoleInternal+0×1b8 0006ee84 7c8018f4 kernel32!ReadConsoleA+0×3b 0006eedc 01005141 kernel32!ReadFile+0×64 0006ef04 01006974 ntsd!ConIn+0×183 0006ff38 010082d1 ntsd!MainLoop+0×1eb 0006ff44 01008589 ntsd!main+0×149 0006ffc0 7c82f23b ntsd!mainCRTStartup+0×12f 0006fff0 00000000 kernel32!BaseProcessStart+0×23

We follow LPC chain to csrss.exe to find out another blocked thread there:

0: kd> !lpc message 00001f98 Searching message 1f98 in threads … Client thread 859f8768 waiting a reply from 1f98 Searching thread 859f8768 in port rundown queues …

Server communication port 0xe19b6b08 Handles: 1 References: 1 The LpcDataInfoChainHead queue is empty Connected port: 0xe19f03a0 Server connection port: 0xe1361d20

Client communication port 0xe19f03a0 Handles: 1 References: 4 The LpcDataInfoChainHead queue is empty

Server connection port e1361d20 Name: ServiceAPort Handles: 1 References: 233 Server process : 85e4fd88 (csrss.exe) Queue semaphore : 85e9b078 Semaphore state 0 (0×0) The message queue is empty The LpcDataInfoChainHead queue is empty Done.

0: kd> !process 85e4fd88 ff […] THREAD 8549db60 Cid 01cc.1390 Teb: 7ffad000 Win32Thread: bc15aea8 WAIT: (Unknown) UserMode Non-Alertable 8549dd4c Semaphore Limit 0×1 Waiting for reply to LPC MessageId 00004feb: Pending LPC Reply Message: e191b6d0: [e1a162e8,e19ffc18] Not impersonating DeviceMap e10008e8 Owning Process 85e4fd88 Image: csrss.exe Attached Process N/A Image: N/A Wait Start TickCount 12095 Ticks: 84698 (0:00:22:03.406) Context Switch Count 35 LargeStack UserTime 00:00:00.000 KernelTime 00:00:00.000 Start Address 0×75943b55 Stack Init f5625000 Current f5624bf0 Base f5625000 Limit f5622000 Call 0 Priority 15 BasePriority 13 PriorityDecrement 0 Kernel stack not resident. ChildEBP RetAddr f5624c08 80833ec5 nt!KiSwapContext+0×26 f5624c34 80829c14 nt!KiSwapThread+0×2e5 f5624c7c 809222f6 nt!KeWaitForSingleObject+0×346 f5624d38 8088b3fc nt!NtSecureConnectPort+0×6ce f5624d38 7c94860c nt!KiFastCallEntry+0xfc 015ff778 7c947939 ntdll!KiFastSystemCallRet 015ff77c 77c2e7c3 ntdll!NtSecureConnectPort+0xc 015ff8a0 77c4607b RPCRT4!LRPC_CASSOCIATION::OpenLpcPort+0×21e 015ff8e0 77c45ffb RPCRT4!LRPC_CASSOCIATION::ActuallyDoBinding+0×55 015ff958 77c4f6a5 RPCRT4!LRPC_CASSOCIATION::AllocateCCall+0×190 015ff98c 77c4f5d1 RPCRT4!LRPC_BINDING_HANDLE::AllocateCCall+0×1f2 015ff9b8 77c4f201 RPCRT4!LRPC_BINDING_HANDLE::NegotiateTransferSyntax+0xd3 015ff9d0 77c4ed14 RPCRT4!I_RpcGetBufferWithObject+0×5b 015ff9e0 77c4f464 RPCRT4!I_RpcGetBuffer+0xf 015ff9f0 77cb30e4 RPCRT4!NdrGetBuffer+0×2e 015ffddc 779b4695 RPCRT4!NdrClientCall2+0×197 […]

We follow LPC chain again to see that csrss.exe thread was waiting for a reply from our suspended and frozen ServiceA:

0: kd> !lpc message 00004feb Searching message 4feb in threads … Client thread 8549db60 waiting a reply from 4feb Searching thread 8549db60 in port rundown queues …

Server connection port e19a50e0 Name: ServiceAPort Handles: 1 References: 20 Server process : 8545eb18 (ServiceA.exe) Queue semaphore : 85443320 Semaphore state 9 (0×9) Messages in queue: 0000 e1a866e0 - Busy Id=000022e7 From: 01e4.01e8 Context=80060004 [e19a50f0 . e1878688] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e1878688 - Busy Id=00003297 From: 0ac0.0b54 Context=804d0045 [e1a866e0 . e1036740] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e1036740 - Busy Id=00003986 From: 0ce4.0ce8 Context=00000042 [e1878688 . e1441228] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e1441228 - Busy Id=00003a32 From: 0db4.0e14 Context=00000050 [e1036740 . e1a162e8] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e1a162e8 - Busy Id=00004c75 From: 059c.05ac Context=00000051 [e1441228 . e191b6d0] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e191b6d0 - Busy Id=00004feb From: 01cc.1390 Context=00000051 [e1a162e8 . e19ffc18] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e19ffc18 - Busy Id=000055e3 From: 13fc.05b4 Context=800d0009 [e191b6d0 . e19f4ea0] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 0000 e19f4ea0 - Busy Id=00006844 From: 0b00.0f20 Context=006b3d60 [e19ffc18 . e19a50f0] Length=011800e8 Type=0000000a (LPC_CONNECTION_REQUEST) Data: 00000000 00000000 00000000 00000000 00000000 00000000 The message queue contains 8 messages The LpcDataInfoChainHead queue is empty Done.

It doesn’t look as a deadlock because, although we have a cyclic process wait chain ServiceA -> NTSD -> CSRSS -> ServiceA, NTSD was waiting for a different thread in CSRSS than the one in CSRSS waiting for a reply from ServiceA. If these threads are unrelated then we don’t have a deadlock, strictly speaking, because the latter involves activity chains with ownership, not a container dependency (a process is a container for threads). I illustrated all this on the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | 1 Comment »

Memory Dumps as Posets

Sunday, August 9th, 2009

Last week I was comparing the existing collection of memory dump analysis patterns to the collection of trace analysis patterns (in formation) in the search of isomorphism (or more correctly, general morphism) similar to Missing Component pattern. It is not a coincidence that such pattern pairs can be formed. For example, it is possible to discern deadlocks from both crash dumps and software traces (if appropriate information is available there). Fundamentally, it is implied by the definition of a software trace as some sort of a memory dump. And we can see traces in memory dumps too, for example, Execution Residue pattern. Because raw stack data resides in stack pages and in contemporary operating systems they are created from zero pages (metaphorically, out of the void) we can say that stack regions of threads are sorted by their creation time, for example, in this process user memory dump:

0:017> !runaway 4 Elapsed Time Thread Time 0:49c 0 days 5:16:31.076 4:4d8 0 days 5:16:30.967 3:4d0 0 days 5:16:30.967 2:4cc 0 days 5:16:30.967 1:4c8 0 days 5:16:30.967 5:4e8 0 days 5:16:30.936 6:b6c 0 days 5:16:15.695 7:b70 0 days 5:16:15.679 9:b88 0 days 5:16:15.586 8:b84 0 days 5:16:15.586 11:348 0 days 5:16:12.934 10:bfc 0 days 5:16:12.934 12:1200 0 days 5:15:16.528 15:1298 0 days 5:15:15.220 14:1290 0 days 5:15:15.220 13:128c 0 days 5:15:15.220 17:12e4 0 days 5:15:13.257 16:12dc 0 days 5:15:13.257 18:12ec 0 days 5:15:13.117 20:12f4 0 days 5:15:13.085 19:12f0 0 days 5:15:13.085 21:17a0 0 days 5:13:16.321 22:1628 0 days 5:13:15.729 24:1778 0 days 1:35:50.773 23:17ec 0 days 1:35:50.773 25:1570 0 days 1:27:54.190 26:1724 0 days 1:27:10.151 27:1490 0 days 0:05:46.732 28:1950 0 days 0:02:28.153 29:19b4 0 days 0:00:58.108 30:177c 0 days 0:00:38.358 31:1798 0 days 0:00:23.351 32:1a7c 0 days 0:00:08.343

If we have complete memory dumps we can also account for other processes and their elapsed time. Within stack pages we have partial stack traces but do not have exact timing information between them except for stack frames from the current frozen thread stack trace or, if we are lucky, from a partial stack trace from the past execution. However, the timing between frames from different stacks is undefined and we can only guess it from higher level considerations like semantics of procedure calls and other information.

These considerations and the notion of a poset (partially ordered set) let me thinking about memory dumps as posets. I even created my interpretation of POSET abbreviation for this occasion:

POSET

Partially Ordered Software Execution Trace

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Mathematics of Debugging, New Acronyms, Science of Memory Dump Analysis, Science of Software Tracing, Software Trace Analysis | No Comments »

Moving towards the Psi point

Thursday, August 6th, 2009

The hierarchy of Ψ₁, …, Ψ₈, …, Ψ₁₆, …, Ψ₃₂, …, Ψ₆₄, …, …, …, Ψ_Ψ numbers where the subscript denotes the number of bits a memory address can have, so Ψ₃₂ and Ψ₆₄ are memorillion and quadrimemorillion of memory dumps respectively. We only need to figure out the meaning of Ψ₀ and Ψ_Ψ. Perhaps there is some meaning in Dirac notation here: <Ψ₀|Ψ_Ψ>. More on this later because I have to finish this week the book x64 Windows Debugging: Practical Foundations and write an errata file for the previous x86 version of the book series.

Note: Ψ is an M upside down.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Fun with Crash Dumps, Futuristic Memory Dump Analysis, Intelligent Memory Movement, Psi-computation, Science of Memory Dump Analysis | No Comments »

NULL data pointer, stack trace, inline function optimization and platformorphic fault: pattern cooperation

Monday, July 27th, 2009

We have the following crash pointing to Driver.sys

7: kd> KL Child-SP RetAddr Call Site fffffadd`7671a678 fffff800`0102e5f4 nt!KeBugCheckEx fffffadd`7671a680 fffff800`0102d587 nt!KiBugCheckDispatch+0x74 fffffadd`7671a800 fffffadd`88e5dbf3 nt!KiPageFault+0x207 fffffadd`7671a998 fffffadd`88df63f5 Driver!memcpy+0×83 fffffadd`7671a9a0 fffffadd`88dfe97b Driver!ItemCopyTo+0×85 fffffadd`7671a9e0 fffffadd`88e45bd1 Driver!CallbackEx+0×3cb fffffadd`7671aa80 fffffadd`88dfb130 Driver!Callback+0×131 fffffadd`7671ab90 fffffadd`88dfaef3 Driver!Reply+0×1a0 fffffadd`7671ac40 fffffadd`88de9e23 Driver!OnDataReceive+0×1a3 fffffadd`7671acc0 fffff800`0124e932 Driver!ReaderThread+0×553 fffffadd`7671ad70 fffff800`010202b6 nt!PspSystemThreadStartup+0×3e fffffadd`7671add0 00000000`00000000 nt!KxStartSystemThread+0×16

7: kd> !analyze -v [...] DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1) An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses. If kernel debugger is available get stack backtrace. Arguments: Arg1: 0000000000000007, memory referenced Arg2: 0000000000000002, IRQL Arg3: 0000000000000001, value 0 = read operation, 1 = write operation Arg4: fffffadd88e5dbf3, address which referenced memory [...] TRAP_FRAME: fffffadd7671a800 -- (.trap 0xfffffadd7671a800) [...]

7: kd> .trap 0xfffffadd7671a800 [...] NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect

7: kd> r Last set context: rax=0000000000001000 rbx=0000000000000000 rcx=0000000000000007 rdx=fffffadda17d001d rsi=0000000000000000 rdi=0000000000000000 rip=fffffadd88e5dbf3 rsp=fffffadd7671a998 rbp=0000000000000001 r8=0000000000000001 r9=fffffadda254f2f0 r10=0000000000000000 r11=0000000000000007 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz na pe nc cs=0010 ss=0018 ds=0006 es=0000 fs=fadf gs=ffff efl=00010202 Driver!memcpy+0×83: fffffadd`88e5dbf3 8801 mov byte ptr [rcx],al ds:0006:0007=??

.trap warning above perhaps explains why we have non-standard values in ds, fs and gs. Typical expected values in kernel mode are these (ds is ignored in 64-bit mode anyway):

cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202

7: kd> kL Child-SP RetAddr Call Site fffffadd`7671a998 fffffadd`88df63f5 Driver!memcpy+0×83 fffffadd`7671a9a0 fffffadd`88dfe97b Driver!ItemCopyTo+0×85 fffffadd`7671a9e0 fffffadd`88e45bd1 Driver!CallbackEx+0×3cb fffffadd`7671aa80 fffffadd`88dfb130 Driver!Callback+0×131 fffffadd`7671ab90 fffffadd`88dfaef3 Driver!Reply+0×1a0 fffffadd`7671ac40 fffffadd`88de9e23 Driver!OnDataReceive+0×1a3 fffffadd`7671acc0 fffff800`0124e932 Driver!ReaderThread+0×553 fffffadd`7671ad70 fffff800`010202b6 nt!PspSystemThreadStartup+0×3e fffffadd`7671add0 00000000`00000000 nt!KxStartSystemThread+0×16

We clearly have an instance of a NULL pointer data access. If we try to match this stack trace to known faults in database we would probably find many entries because memcpy is a generic function from C library. So we should try with ItemCopyTo. Indeed, we find a few matches but with slightly different stack traces:

b7535c7c b75931fa Driver!ItemCopyTo+0×6a b7535ca4 b75c24c4 Driver!CallbackEx+0×23a b7535d04 b7590c79 Driver!Callback+0xd4 b7535d44 b7590b41 Driver!Reply+0xe9 b7535d68 b7584b87 Driver!OnDataReceive+0×111 b7535dac 8094bea4 Driver!ReaderThread+0×397 b7535ddc 8088f61e nt!PspSystemThreadStartup+0×2e 00000000 00000000 nt!KiThreadStartup+0×16

Offsets are different but the function names are the same. We also don’t see memcpy call but if we look at the faulted instruction we suspect it was inlined memcpy call:

eax=88642870 ebx=00000005 ecx=00000001 edx=00000005 esi=88f3f9a4 edi=00000029 eip=b758d3ca esp=b7535c6c ebp=b7535c7c iopl=0 nv up ei pl nz ac po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010212 Driver!ItemCopyTo+0×6a: b758d3ca f3a5 rep movs dword ptr es:[edi],dword ptr [esi]

We also notice that the found stack trace is from x86 32-bit Windows but ours is from x64 Windows so we suspect the platformorphic fault here and check if we have a fix for x64 binaries.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

On Self Dumps of Secure String API

Tuesday, July 14th, 2009

Sometimes we get crashes with the following stack trace pattern:

0:000> kv 100 ChildEBP RetAddr Args to Child 0012cecc 7c91df4a 7c8094fe 00000002 0012cef8 ntdll!KiFastSystemCallRet 0012ced0 7c8094fe 00000002 0012cef8 00000001 ntdll!ZwWaitForMultipleObjects+0xc 0012cf6c 7c80a085 00000002 0012d09c 00000000 kernel32!WaitForMultipleObjectsEx+0x12c 0012cf88 6990763c 00000002 0012d09c 00000000 kernel32!WaitForMultipleObjects+0x18 0012d91c 699082b1 0012f090 00000001 00198312 faultrep!StartDWException+0×5df 0012e990 7c8635d1 0012f090 00000001 00000000 faultrep!ReportFault+0×533 0012f030 78138a09 0012f090 00000022 c000000d kernel32!UnhandledExceptionFilter+0×587 0012f368 781443d1 00000000 00000000 00000000 msvcr80!_invoke_watson+0xc4 0012f38c 0040b02f 0012f538 00000104 004f80a0 msvcr80!strcat_s+0×29 WARNING: Stack unwind information not available. Following frames may be wrong. 0012f39c 0012f538 00000104 00430848 0012f538 Application+0xb020

0:000> .exptr 0012f090

----- Exception record at 0012f040: ExceptionAddress: 781443d1 (msvcr80!strcat_s+0x00000029) ExceptionCode: c000000d ExceptionFlags: 00000000 NumberParameters: 0

----- Context record at 0012f098: eax=f2c4dacf ebx=00000000 ecx=00000002 edx=7c91e514 esi=00000022 edi=00000000 eip=781443d1 esp=0012f36c ebp=004f7658 iopl=0 nv up ei ng nz na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000286 msvcr80!strcat_s+0x29: 781443d1 83c414 add esp,14h

0:000> !error c000000d Error code: (NTSTATUS) 0xc000000d (3221225485) - An invalid parameter was passed to a service or function.

We clearly see that the crash involved strcat_s but we don’t see any invalid instruction or access violation call. The ADD instruction is perfectly valid above. However, if we disassemble EIP backwards we would see the call to _invalid_parameter function and it looks like it has a name association with Dr. Watson:

0:000> ub 781443d1 msvcr80!strcat_s+0×1c: 781443c4 5e pop esi 781443c5 8930 mov dword ptr [eax],esi 781443c7 53 push ebx 781443c8 53 push ebx 781443c9 53 push ebx 781443ca 53 push ebx 781443cb 53 push ebx 781443cc e89f46ffff call msvcr80!_invalid_parameter (78138a70)

0:000> uf 78138a70 Flow analysis was incomplete, some code may be missing msvcr80!_invoke_watson: 78138945 55 push ebp 78138946 8dac2458fdffff lea ebp,[esp-2A8h] 7813894d 81ec28030000 sub esp,328h

[...]

msvcr80!_invalid_parameter: 78138a70 55 push ebp 78138a71 8bec mov ebp,esp 78138a73 ff3528401c78 push dword ptr [msvcr80!__pInvalidArgHandler (781c4028)] 78138a79 e85ba1ffff call msvcr80!_decode_pointer (78132bd9) 78138a7e 85c0 test eax,eax 78138a80 59 pop ecx 78138a81 7403 je msvcr80!_invalid_parameter+0×16 (78138a86)

msvcr80!_invalid_parameter+0x13: 78138a83 5d pop ebp 78138a84 ffe0 jmp eax

msvcr80!_invalid_parameter+0x16: 78138a86 6a02 push 2 78138a88 e806330000 call msvcr80!_crt_debugger_hook (7813bd93) 78138a8d 59 pop ecx 78138a8e 5d pop ebp 78138a8f e9b1feffff jmp msvcr80!_invoke_watson (78138945)

According to MSDN documentation, _s secure functions by default use a postmortem debugger mechanism:

Secure-enhanced CRT parameter validation

So we have something that is similar to Self-Dump pattern here. The same parameter checking is seen in the case of C++ STL exceptions. In case of custom unhandled exception filters not resorting to WER faulrep.dll other stacks can show process termination, for example, with wcscpy_s:

0:000> kL 100 ChildEBP RetAddr 0111cb64 7c947c39 ntdll!KiFastSystemCallRet 0111cb68 7c80202b ntdll!ZwTerminateProcess+0xc 0111cb78 78138a2b kernel32!TerminateProcess+0×20 0111ceb4 78144ba1 MSVCR80!_invoke_watson+0xe6 0111ced8 67dbb47d MSVCR80!wcscpy_s+0×29 0111cf00 67dbc93b Application!CopyName+0×5d […]

Here the specified size of the destination buffer was smaller than the size of source NULL-terminated strings, which was a good thing anyway: old strcpy or strcat function would definitely caused buffer overflow effects. Now we have a nice side effect too, the dump is saved, ready for a postmortem analysis and subsequent code improvement.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Assembly Language, Crash Dump Analysis, Crash Dump Patterns, Debugging, Security | No Comments »

WOW64 process, NULL data pointer, stack overflow, main thread, incorrect stack trace, nested exceptions, hidden exception, manual dump, multiple exceptions and virtualized system: pattern cooperation

Sunday, July 12th, 2009

One 32-bit WOW64 process was crashing when accessing a direct NULL data pointer with the following stack trace:

0:000> r rax=00000000750e9b40 rbx=000000000017f29c rcx=0000000000041710 rdx=0000000000041240 rsi=0000000000041710 rdi=0000000000041240 rip=00000000750e9b59 rsp=00000000000411f0 rbp=000000000017f2a8 r8=0000000000000001 r9=00000000750ffd40 r10=0000000000000000 r11=000000000017f29c r12=000000007efdb000 r13=000000000007fd20 r14=000000000007ee70 r15=00000000751e3380 iopl=0 nv up ei pl nz na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 wow64!Wow64PrepareForException+0×19: 00000000`750e9b59 654c8b1c2530000000 mov r11,qword ptr gs:[30h] gs:00000000`00000030=????????????????

0:000> kc Call Site wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher

It looks like a stack overflow. Usually it manifests via a PUSH instruction or a data access violation when ESP/RSP < TEB.StackLimit. However here RSP is still inside the stack range:

0:000> !teb [...] Wow64 TEB at 000000007efdb000 ExceptionList: 000000007efdd000 StackBase: 000000000007fd20 StackLimit: 0000000000041000 SubSystemTib: 0000000000000000 FiberData: 0000000000001e00 ArbitraryUserPointer: 0000000000000000 Self: 000000007efdb000 EnvironmentPointer: 0000000000000000 ClientId: 0000000000001684 . 000000000000168c RpcHandle: 0000000000000000 Tls Storage: 0000000000000000 PEB Address: 000000007efdf000 LastErrorValue: 0 LastStatusValue: c0000034 Count Owned Locks: 0 HardErrorMode: 0

0:000> dq rsp-10 rsp+10 00000000`000411e0 00000000`00000000 00000000`00000000 00000000`000411f0 00000000`00000000 00000000`00000000 00000000`00041200 00000000`00000000

But still RSP is very close to the stack limit and the next function call surely would overflow because of the large frame delta:

0:000> kcf Memory Call Site wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher 5a0 wow64!Wow64PrepareForException 50 ntdll!KiUserExceptionDispatcher

So we consider this as the stack overflow indeed. We specify the larger number of frames to reach the stack base:

0:000> kc 1000 Call Site wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher [...] ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64PrepareForException ntdll!KiUserExceptionDispatcher wow64!Wow64SystemServiceEx wow64cpu!ServiceNoTurbo wow64!RunCpuSimulation wow64!Wow64LdrpInitialize ntdll!LdrpInitializeProcess ntdll! ?? ::FNODOBFM::`string’ ntdll!LdrInitializeThunk

We consider this as a nested unmanaged exception and try to see where it originated. First we double check that we don’t have any exceptions in 32-bit code. And indeed threads look clean:

0:000:x86> ~*kc

. 0 Id: 1684.168c Suspend: 0 Teb: 7efdb000 Unfrozen

user32!NtUserGetProp user32!GetPropA WARNING: Stack unwind information not available. Following frames may be wrong. Application!foo user32!InternalCallWinProc user32!UserCallWinProcCheckWow user32!SendMessageWorker user32!SendMessageA Application

1 Id: 1684.1688 Suspend: 1 Teb: 7efd8000 Unfrozen

ntdll_77320000!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus rpcrt4!COMMON_ProcessCalls rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents rpcrt4!ProcessIOEventsWrapper rpcrt4!BaseCachedThreadRoutine rpcrt4!ThreadStartRoutine kernel32!BaseThreadInitThunk ntdll_77320000!__RtlUserThreadStart ntdll_77320000!_RtlUserThreadStart

2 Id: 1684.1678 Suspend: 1 Teb: 7efd5000 Unfrozen

ntdll_77320000!NtDelayExecution kernel32!SleepEx kernel32!Sleep ole32!CROIDTable::WorkerThreadLoop ole32!CRpcThread::WorkerLoop ole32!CRpcThreadCache::RpcWorkerThreadEntry kernel32!BaseThreadInitThunk ntdll_77320000!__RtlUserThreadStart ntdll_77320000!_RtlUserThreadStart

3 Id: 1684.164c Suspend: 1 Teb: 7efad000 Unfrozen

ntdll_77320000!NtWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects msiltcfg!WorkerThread kernel32!BaseThreadInitThunk ntdll_77320000!__RtlUserThreadStart ntdll_77320000!_RtlUserThreadStart

4 Id: 1684.166c Suspend: 1 Teb: 7efaa000 Unfrozen

ntdll_77320000!ZwWaitForSingleObject kernel32!WaitForSingleObjectEx kernel32!WaitForSingleObject winspool!MonitorRPCServerProcess kernel32!BaseThreadInitThunk ntdll_77320000!__RtlUserThreadStart ntdll_77320000!_RtlUserThreadStart

The first thread looks a bit suspicious, we have never seen NtUserGetProp on stack traces, there is a possibility of an exception in main GUI thread and also the stack trace itself looks incorrect, suddenly sending a Windows message without any kind of a message loop:

0:000:x86> k ChildEBP RetAddr 0017f294 76e45be7 user32!NtUserGetProp+0×15 0017f2a8 025d4ba0 user32!GetPropA+0×3d WARNING: Stack unwind information not available. Following frames may be wrong. 0017f2e0 76e38807 Application!foo+0×230 0017f30c 76e38962 user32!InternalCallWinProc+0×23 0017f384 76e3c4b6 user32!UserCallWinProcCheckWow+0×109 0017f3c8 76e3eae2 user32!SendMessageWorker+0×55b 0017f3ec 02a755c6 user32!SendMessageA+0×7f 00000000 00000000 Application+0×255c6

NtUserGetProp EIP address looks valid and points to the code after the return from a system call:

0:000:x86> u eip user32!NtUserGetProp+0x15: 76e3b64f c20800 ret 8 76e3b652 90 nop 76e3b653 90 nop 76e3b654 90 nop 76e3b655 90 nop 76e3b656 90 nop user32!GetPropW: 76e3b657 8bff mov edi,edi 76e3b659 55 push ebp

0:000:x86> ub eip user32!DefWindowProcW+0x96: 76e3b636 90 nop 76e3b637 90 nop 76e3b638 90 nop 76e3b639 90 nop user32!NtUserGetProp: 76e3b63a b80e100000 mov eax,100Eh 76e3b63f 8d542404 lea edx,[esp+4] 76e3b643 b900000000 mov ecx,0 76e3b648 64ff15c0000000 call dword ptr fs:[0C0h]

Its 32-bit raw stack looks like normal main GUI thread of a VB6 application:

0:000:x86> !wow64exts.info [...] 32 bit, StackBase : 0×180000 StackLimit : 0×17c000 Deallocation: 0×80000

64 bit, StackBase : 0x7fd20 StackLimit : 0x41000 Deallocation: 0x40000 [...]

0:000:x86> dds 0x17c000 0x180000 0017c000 00000000 0017c004 00000000 0017c008 00000000 0017c00c 00000000 0017c010 00000000 [...] 0017fdc8 00000000 0017fdcc 0017fe30 0017fdd0 76e38aad user32!DispatchMessageWorker+0×380 0017fdd4 00000000 0017fdd8 726ff5d1 msvbvm60!IID_IVbaHost+0×30ff1 0017fddc 0002088c 0017fde0 00000113 0017fde4 0002088c 0017fde8 00000000 0017fdec 00b319a8 0017fdf0 00000001 0017fdf4 9f7e168b 0017fdf8 76e395c0 user32!PeekMessageA 0017fdfc 02c0209c 0017fe00 00000000 0017fe04 76e395c0 user32!PeekMessageA 0017fe08 02c0216c 0017fe0c 0002088c 0017fe10 00000000 0017fe14 726aa76a msvbvm60!_vbaStrToAnsi+0×3ab 0017fe18 0017fdf4 0017fe1c ffffffff 0017fe20 0017ff70 0017fe24 76e985e7 user32!_except_handler4 0017fe28 e98a626b 0017fe2c fffffffe 0017fe30 0017fe40 0017fe34 76e39100 user32!DispatchMessageA+0xf […] 0017ff58 ffffffff 0017ff5c ffffffff 0017ff60 ffffffff 0017ff64 00000000 0017ff68 0017ff10 0017ff6c 00000000 0017ff70 0017ffc4 0017ff74 7278bafd msvbvm60!CreateIExprSrvObj+0×9f2 0017ff78 726b97d0 msvbvm60!BASIC_CLASS_Release+0xadbd 0017ff7c 00000000 0017ff80 0017ff94 0017ff84 00401396 0017ff88 00401514 0017ff8c 768de3f3 kernel32!BaseThreadInitThunk+0xe 0017ff90 7efde000 0017ff94 0017ffd4 0017ff98 7739cfed ntdll_77320000!__RtlUserThreadStart+0×23 0017ff9c 7efde000 0017ffa0 63553bf6 0017ffa4 00000000 0017ffa8 00000000 0017ffac 7efde000 0017ffb0 00000000 0017ffb4 00000000 0017ffb8 00000000 0017ffbc 0017ffa0 0017ffc0 00000000 0017ffc4 0017ffe4 0017ffc8 773d2926 ntdll_77320000!_except_handler4 0017ffcc 14761e1a 0017ffd0 00000000 0017ffd4 0017ffec 0017ffd8 7739d1ff ntdll_77320000!_RtlUserThreadStart+0×1b 0017ffdc 0040138c 0017ffe0 7efde000 0017ffe4 ffffffff 0017ffe8 773bd377 ntdll_77320000!FinalExceptionHandler 0017ffec 00000000 0017fff0 00000000 0017fff4 0040138c 0017fff8 7efde000 0017fffc 00000000 00180000 78746341

There are no any signs of exception codes and processing residue there and we come back to out 64-bit layer:

0:000:x86> .effmach amd64 Effective machine: x64 (AMD64)

We dump 64-bit stack to see the moment when the first exception happened:

0:000> dqs 0x41000 0x7fd20 [...] 00000000`0007df30 00000000`00000000 00000000`0007df38 00000000`00000000 00000000`0007df40 00000000`00000000 00000000`0007df48 00000000`00000000 00000000`0007df50 00000000`00000000 00000000`0007df58 00000000`00000000 00000000`0007df60 00000000`00000000 00000000`0007df68 00000000`00000000 00000000`0007df70 00000000`00000000 00000000`0007df78 00000000`00000000 00000000`0007df80 00000000`750e9b59 wow64!Wow64PrepareForException+0x19 00000000`0007df88 00000000`00000000 00000000`0007df90 00000000`00000000 00000000`0007df98 00000000`0007dfb0 00000000`0007dfa0 00000000`00000001 00000000`0007dfa8 00000000`0007e510 00000000`0007dfb0 00000000`611b422e 00000000`0007dfb8 fffffa60`04eec978 00000000`0007dfc0 fffffa60`04ee6520 00000000`0007dfc8 fffffa60`04ee6520 00000000`0007dfd0 fffffa60`04eec978 00000000`0007dfd8 00000000`00000030 00000000`0007dfe0 00000000`0017f29c 00000000`0007dfe8 00000000`0000c04c 00000000`0007dff0 00000000`00000000 00000000`0007dff8 00000000`771c59e6 ntdll!KiUserExceptionDispatcher+0×1c 00000000`0007e000 01c9f89c`fe787c8d 00000000`0007e008 fffffa60`03732e4e 00000000`0007e010 fffffa80`0497c9a0 00000000`0007e018 00000000`00000000 00000000`0007e020 00000000`00000001 00000000`0007e028 01c9f89c`fe787c00 00000000`0007e030 00001fa0`0010001f 00000000`0007e038 0053002b`002b0033 00000000`0007e040 00010283`002b002b 00000000`0007e048 00000000`00000000 00000000`0007e050 00000000`00000000 00000000`0007e058 00000000`00000000 00000000`0007e060 00000000`00000000 00000000`0007e068 00000000`00000000 00000000`0007e070 00000000`00000000 00000000`0007e078 00000000`00000006 00000000`0007e080 00000000`0000100e 00000000`0007e088 00000000`0000000e 00000000`0007e090 00000000`0017f29c 00000000`0007e098 00000000`0007e5a0 00000000`0007e0a0 00000000`0017f2a8 00000000`0007e0a8 00000000`0000c04c 00000000`0007e0b0 00000000`00000000 00000000`0007e0b8 00000000`00000001 00000000`0007e0c0 00000000`750ffd40 wow64!ServiceTables 00000000`0007e0c8 00000000`00000000 00000000`0007e0d0 00000000`0017f29c 00000000`0007e0d8 00000000`7efdb000 00000000`0007e0e0 00000000`0007fd20 00000000`0007e0e8 00000000`0007ee70 00000000`0007e0f0 00000000`751e3380 wow64cpu!CpupSaveLegacyFloatingPointState+0×60 00000000`0007e0f8 00000000`750ca923 wow64!Wow64SystemServiceEx+0×57 00000000`0007e100 015c0000`4000137f 00000000`0007e108 00000023`02add97f 00000000`0007e110 0000002b`0017f408 00000000`0007e118 0000ffff`00001f80 00000000`0007e120 00000000`00000000 00000000`0007e128 00000000`00000000

All addresses we used to try for .exr and .cxr commands in hidden exception pattern are beyond user space and we therefore conclude that somehow such structures or pointers to them became corrupt and their access triggered the same exception processing code over and over again finally leading to the stack overflow. This was an x64 system and taking a complete memory dump was impractical so at the same time when a WER error message appeared a kernel memory dump was generated (a pair of dumps from fiber bundle) and it shows the final stages of exception processing that are in agreement with the user dump we analyzed.

Initially we thought that the system experienced a bugcheck after the application crash:

1: kd> !analyze -v DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1) An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses. If kernel debugger is available get stack backtrace. Arguments: Arg1: fffff880058b1010, memory referenced Arg2: 0000000000000002, IRQL Arg3: 0000000000000000, value 0 = read operation, 1 = write operation Arg4: fffffa60053da17a, address which referenced memory

But then we noticed that bugcheck was forced intentionally using NotMyFault sysinternals tool:

1: kd> kc Call Site nt!KeBugCheckEx nt!KiBugCheckDispatch nt!KiPageFault myfault myfault nt!IopXxxControlFile nt!NtDeviceIoControlFile nt!KiSystemServiceCopyEnd 0×0

In the kernel dump we see our application and the next process is WER:

1: kd> !vm [...] PROCESS fffffa8001886710 SessionId: 1 Cid: 13e4 Peb: 7efdf000 ParentCid: 0ab0 DirBase: 329c8000 ObjectTable: fffff880059424d0 HandleCount: 190. Image: Application.exe

PROCESS fffffa800188a040 SessionId: 1 Cid: 0b44 Peb: 7efdf000 ParentCid: 0934 DirBase: 2b144000 ObjectTable: fffff88004c68a30 HandleCount: 166. Image: WerFault.exe

We see a page fault on a kernel stack trace of the main process thread:

1: kd> !thread fffffa8001cc7bb0 1f THREAD fffffa8001cc7bb0 Cid 13e4.01dc Teb: 000000007efdb000 Win32Thread: fffff900c210ed50 WAIT: (WrLpcReply) UserMode Non-Alertable fffffa8001cc7f40 Semaphore Limit 0x1 Waiting for reply to ALPC Message fffff880075c3970 : queued at port fffffa8001dae290 : owned by process fffffa8001d8e040 Not impersonating DeviceMap fffff88004c30a00 Owning Process fffffa8001886710 Image: Application.exe Attached Process N/A Image: N/A Wait Start TickCount 83669865 Ticks: 320 (0:00:00:04.992) Context Switch Count 1829 LargeStack UserTime 00:00:00.187 KernelTime 00:00:00.280 Win32 Start Address 0x000000000040138c Stack Init fffffa6004cc3db0 Current fffffa6004cc2ae0 Base fffffa6004cc4000 Limit fffffa6004cb9000 Call 0 Priority 10 BasePriority 8 PriorityDecrement 2 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffffa60`04cc2b20 fffff800`0165cf8a nt!KiSwapContext+0×7f fffffa60`04cc2c60 fffff800`0165e38a nt!KiSwapThread+0×2fa fffffa60`04cc2cd0 fffff800`0168ecab nt!KeWaitForSingleObject+0×2da fffffa60`04cc2d60 fffff800`018e1f64 nt!AlpcpSignalAndWait+0×7b fffffa60`04cc2da0 fffff800`018e7216 nt!AlpcpReceiveSynchronousReply+0×44 fffffa60`04cc2e00 fffff800`018dc27f nt!AlpcpProcessSynchronousRequest+0×24f fffffa60`04cc2f20 fffff800`01656e33 nt!NtAlpcSendWaitReceivePort+0×19f fffffa60`04cc2fd0 fffff800`01657340 nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`04cc3040) fffffa60`04cc31d8 fffff800`019b6e30 nt!KiServiceLinkage fffffa60`04cc31e0 fffff800`01898d6d nt!DbgkpSendErrorMessage+0×230 fffffa60`04cc3300 fffff800`016bf487 nt!DbgkForwardException+0×12d fffffa60`04cc3440 fffff800`016571e9 nt! ?? ::FNODOBFM::`string’+0×29550 fffffa60`04cc3a40 fffff800`01655fe5 nt!KiExceptionDispatch+0xa9 fffffa60`04cc3c20 00000000`75319b59 nt!KiPageFault+0×1e5 (TrapFrame @ fffffa60`04cc3c20) 00000000`000411f0 00000000`00000000 0×75319b59

Analysis of the raw stack confirms the final access violation address:

1: kd> dqs fffffa6004cb9000 fffffa6004cc4000 […] fffffa60`04cc3a00 00000000`00000000 fffffa60`04cc3a08 00000000`00000000 fffffa60`04cc3a10 00000000`00000000 fffffa60`04cc3a18 00000000`00000000 fffffa60`04cc3a20 00000000`0007fd20 fffffa60`04cc3a28 00000000`7efdb000 fffffa60`04cc3a30 00000000`00041240 fffffa60`04cc3a38 fffff800`016571e9 nt!KiExceptionDispatch+0xa9 fffffa60`04cc3a40 fffffa60`04cc3b78 fffffa60`04cc3a48 00000000`0017f290 fffffa60`04cc3a50 fffffa60`04cc3c20 fffffa60`04cc3a58 00000000`00041710 fffffa60`04cc3a60 00000000`00000001 fffffa60`04cc3a68 00000000`80000001 fffffa60`04cc3a70 0000002d`0000002d fffffa60`04cc3a78 0000002d`0000002d fffffa60`04cc3a80 0000002d`0000002d fffffa60`04cc3a88 0000002d`0000002d fffffa60`04cc3a90 00000000`00000000 fffffa60`04cc3a98 00000000`00000000 fffffa60`04cc3aa0 00000000`00000000 fffffa60`04cc3aa8 00000000`00000000 fffffa60`04cc3ab0 00000000`00000000 fffffa60`04cc3ab8 00000000`00000000 fffffa60`04cc3ac0 00000000`00000000 fffffa60`04cc3ac8 00000000`00000000 […]

1: kd> .exr fffffa60`04cc3b78 ExceptionAddress: 0000000075319b59 ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 0000000000000000 Parameter[1]: 0000000000000030 Attempt to read from address 0000000000000030

Looking at drivers shows that the system was virtualized under a VM:

1: kd> lm [...] fffffa60`037bc000 fffffa60`037e0000 3rdpartyVM (deferred) [...]

Therefore we suggested to test the application without VM to rule out vitrualization influence.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Virtualization | 6 Comments »

Crash Dump Analysis Patterns (Part 87)

Friday, July 10th, 2009

Sometimes we get rare or hardware-related bugchecks and these might have been influenced by hardware virtualization (that is also a software and could have its own defects). Therefore it is beneficial to recognize when a system is running under a VM (Virtualized System pattern):

Memory Dumps from Xen-virtualized Windows

Memory dumps from VMware images (Virtual PC diagnostics in post comments)

Memory Dumps from Hyper-Virtualized Windows

For example, we get the following bugcheck and stack trace for the first processor:

0: kd> !analyze -v [...] CLOCK_WATCHDOG_TIMEOUT (101) An expected clock interrupt was not received on a secondary processor in an MP system within the allocated interval. This indicates that the specified processor is hung and not processing interrupts. Arguments: Arg1: 00000060, Clock interrupt time out interval in nominal clock ticks. Arg2: 00000000, 0. Arg3: 805d8120, The PRCB address of the hung processor. Arg4: 00000001, 0.

CURRENT_IRQL: 1c

STACK_TEXT: 8d283694 816df8a5 nt!KeBugCheckEx+0x1e 8d2836c8 816e163d nt!KeUpdateRunTime+0xd5 8d2836c8 84617008 nt!KeUpdateSystemTime+0xed WARNING: Frame IP not in any known module. Following frames may be wrong. 8d283748 816f46a1 0×84617008 8d283758 816fa6aa nt!HvlpInitiateHypercall+0×21 8d283784 8166aca5 nt!HvlNotifyLongSpinWait+0×2b 8d2837a0 816cce7e nt!KeFlushSingleTb+0xc4 8d283808 81681db4 nt!MmAccessFault+0xc1d 8d283808 816dd033 nt!KiTrap0E+0xdc 8d28389c 8168ed58 nt!memcpy+0×33 8d283954 816712bf nt!MmCopyToCachedPage+0×1193 8d2839ec 81663053 nt!CcMapAndCopy+0×210 8d283a74 8c688218 nt!CcFastCopyWrite+0×283 8d283b98 8c40badc Ntfs!NtfsCopyWriteA+0×23e 8d283bcc 8c40bcab fltmgr!FltpPerformFastIoCall+0×22e 8d283bf8 8c41dc30 fltmgr!FltpPassThroughFastIo+0×7d 8d283c3c 818471cd fltmgr!FltpFastIoWrite+0×146 8d283d38 8167ec7a nt!NtWriteFile+0×34c 8d283d38 77115e74 nt!KiFastCallEntry+0×12a 01cfee80 00000000 0×77115e74

The thread was servicing a page fault. Notice the gap between KeUpdateSystemTime and HvlpInitiateHypercall. This is normal and consistent code if we look closer:

0: kd> .asm no_code_bytes Assembly options: no_code_bytes

0: kd> uf HvlpInitiateHypercall nt!HvlpInitiateHypercall: 816f4680 push edi 816f4681 push esi 816f4682 push ebx 816f4683 mov eax,dword ptr [esp+10h] 816f4687 mov edx,dword ptr [esp+14h] 816f468b mov ecx,dword ptr [esp+18h] 816f468f mov ebx,dword ptr [esp+1Ch] 816f4693 mov esi,dword ptr [esp+20h] 816f4697 mov edi,dword ptr [esp+24h] 816f469b call dword ptr [nt!HvlpHypercallCodeVa (8176bb8c)] 816f46a1 pop ebx 816f46a2 pop esi 816f46a3 pop edi 816f46a4 ret 18h

0: kd> dp 8176bb8c l1 8176bb8c 84617000

0: kd> uf 84617000 84617000 or eax,80000000h 84617005 vmcall 84617008ret

We have the address of RET instruction (84617008) on the stack trace:

0: kd> kv ChildEBP RetAddr Args to Child 8d283694 816df8a5 00000101 00000060 00000000 nt!KeBugCheckEx+0x1e 8d2836c8 816e163d 84e1521b 000000d1 8d283784 nt!KeUpdateRunTime+0xd5 8d2836c8 8461700884e1521b 000000d1 8d283784 nt!KeUpdateSystemTime+0xed (FPO: [0,2] TrapFrame @ 8d2836d8) WARNING: Frame IP not in any known module. Following frames may be wrong. 8d283748 816f46a1 84e6c900 22728000 8172e28c 0×84617008 8d283758 816fa6aa 00010008 00000000 22728000 nt!HvlpInitiateHypercall+0×21 (FPO: [6,3,0]) 8d283784 8166aca5 22728000 00000000 00000000 nt!HvlNotifyLongSpinWait+0×2b […]

The second processor is busy too:

0: kd> !running

System Processors 3 (affinity mask) Idle Processors 0

Prcbs Current Next 0 8172c920 84e6c900 ................ 1 805d8120 85138030 85a50d78 …………….

0: kd> !thread 85138030 THREAD 85138030 Cid 0564.11c8 Teb: 7ff9f000 Win32Thread: 00000000 RUNNING on processor 1 IRP List: 85ab5d00: (0006,01fc) Flags: 00000884 Mdl: 00000000 85445ab8: (0006,0094) Flags: 00060000 Mdl: 00000000 Not impersonating DeviceMap 98a7d558 Owning Process 84f0d938 Image: Application.exe Attached Process N/A Image: N/A Wait Start TickCount 695643 Ticks: 224 (0:00:00:03.500) Context Switch Count 20 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address 0x705e2679 Stack Init a1d13000 Current a1d10a70 Base a1d13000 Limit a1d10000 Call 0 Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 ChildEBP RetAddr Args to Child 00000000 00000000 00000000 00000000 00000000 0×0

Because we have (Limit, Current, Base) triple for our thread we check its execution residue on kernel raw stack. We find traces of a hypercall too:

0: kd> dds a1d10000 a1d13000 [...] a1d1215c 816fa6da nt!HvlEndSystemInterrupt+0x20 a1d12160 40000070 a1d12164 00000000 a1d12168 81608838 hal!HalEndSystemInterrupt+0x7a a1d1216c 805d8000 a1d12170 a1d12180 a1d12174 81618cc9 hal!HalpIpiHandler+0x189 a1d12178 84f4841b a1d1217c 000000e1 a1d12180 a1d1222c a1d12184 84617008 a1d12188 badb0d00 a1d1218c 00000000 a1d12190 81665699 nt!RtlWalkFrameChain+0x58 a1d12194 816659c4 nt!RtlWalkFrameChain+0x377 a1d12198 11c4649e a1d1219c 00000002 a1d121a0 00000003 a1d121a4 85478444 a1d121a8 0000001f a1d121ac 00000000 a1d121b0 00000000 a1d121b4 816f46a1 nt!HvlpInitiateHypercall+0×21 a1d121b8 8172c800 nt!KiInitialPCR a1d121bc 85138030 a1d121c0 85a72ac0 a1d121c4 8171b437 nt!HvlSwitchVirtualAddressSpace+0×28 a1d121c8 00010001 a1d121cc 00000000 a1d121d0 85a72ac0 a1d121d4 85138030 a1d121d8 8172c802 nt!KiInitialPCR+0×2 a1d121dc 00000000 a1d121e0 85138030 a1d121e4 816fa6da nt!HvlEndSystemInterrupt+0×20 a1d121e8 40000070 a1d121ec 00000000 a1d121f0 81608838 hal!HalEndSystemInterrupt+0×7a a1d121f4 816f46a1 nt!HvlpInitiateHypercall+0×21 a1d121f8 805d8000 a1d121fc 85138030 a1d12200 805dc1e0 a1d12204 8171b437 nt!HvlSwitchVirtualAddressSpace+0×28 a1d12208 00010001 a1d1220c 00000000 a1d12210 00000000 a1d12214 81608468 hal!HalpDispatchSoftwareInterrupt+0×5e a1d12218 00000000 a1d1221c 00000000 a1d12220 00000206 a1d12224 a1d12240 a1d12228 81608668 hal!HalpCheckForSoftwareInterrupt+0×64 a1d1222c 00000002 a1d12230 00000000 a1d12234 816086a8 hal!KfLowerIrql a1d12238 00000000 a1d1223c 00000002 a1d12240 a1d12250 a1d12244 8160870c hal!KfLowerIrql+0×64 a1d12248 00000000 a1d1224c 00000000 a1d12250 a1d12294 a1d12254 816e035a nt!KiSwapThread+0×477 a1d12258 85138030 a1d1225c 851380b8 a1d12260 805d8120 a1d12264 0014d1f8 […]

Looking at raw stack further we can even see that it was processing a page fault too and manually reconstruct its stack trace:

[...] a1d1074c 85aef510 a1d10750 a1d10768 a1d10754 81678976 nt!IofCallDriver+0×63 a1d10758 84c87d50 a1d1075c 85aef510 a1d10760 00000000 a1d10764 84c87d50 a1d10768 a1d10784 a1d1076c 8166d74e nt!IoPageRead+0×172 a1d10770 85138030 a1d10774 84a1352c a1d10778 84a134f8 a1d1077c 84a13538 a1d10780 84c87d50 a1d10784 a1d10840 a1d10788 816abf07 nt!MiDispatchFault+0xd14 a1d1078c 00000043 a1d10790 85138030 a1d10794 84a13538 a1d10798 84a1350c a1d1079c 84a1352c a1d107a0 8174c800 nt!MmSystemCacheWs a1d107a4 00000000 a1d107a8 85138030 a1d107ac a5397bf8 a1d107b0 85b01c48 a1d107b4 00000000 a1d107b8 00000000 a1d107bc a5397bf8 a1d107c0 84a1358c a1d107c4 a1d10864 a1d107c8 00000000 a1d107cc 8463a590 a1d107d0 84a134f8 a1d107d4 c0518000 a1d107d8 00000000 a1d107dc 00000000 a1d107e0 00000028 a1d107e4 a1d107f4 a1d107e8 00000000 a1d107ec 00000038 a1d107f0 859f5640 a1d107f4 a4bfa390 a1d107f8 00000000 a1d107fc 00000000 a1d10800 00000000 a1d10804 a1d10938 a1d10808 818652bd nt!MmCreateSection+0×98f a1d1080c 00000000 a1d10810 846652e8 a1d10814 00000000 a1d10818 00000000 a1d1081c 00000000 a1d10820 00000028 a1d10824 00000000 a1d10828 00000080 a1d1082c 0000000a a1d10830 85ae1c98 a1d10834 85ae1c20 a1d10838 00000000 a1d1083c 00000000 a1d10840 a1d108b8 a1d10844 816cd325 nt!MmAccessFault+0×10c6 a1d10848 a3000000 a1d1084c a5397bf8 a1d10850 00000000 a1d10854 8174c800 nt!MmSystemCacheWs a1d10858 00000000 a1d1085c 00000000 a1d10860 a5397bf8 a1d10864 00000000 […]

0: kd> k L=a1d10750 a1d10750 a1d10750 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. a1d10750 81678976 0xa1d10750 a1d10768 8166d74e nt!IofCallDriver+0x63 a1d10784 816abf07 nt!IoPageRead+0x172 a1d10840 816cd325 nt!MiDispatchFault+0xd14 a1d108b8 816f0957 nt!MmAccessFault+0x10c6 a1d10924 8181c952 nt!MmCheckCachedPageState+0x801 a1d109b0 8c60f850 nt!CcCopyRead+0x435 a1d109dc 8c613c52 Ntfs!NtfsCachedRead+0x13b a1d10abc 8c612b6f Ntfs!NtfsCommonRead+0x105a a1d10b2c 81678976 Ntfs!NtfsFsdRead+0x273 a1d10b44 8c40cba7 nt!IofCallDriver+0x63 a1d10b68 8c40d7c7 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x251 a1d10ba0 8c40dbe7 fltmgr!FltPerformSynchronousIo+0xb9 a1d10c10 9ca680e5 fltmgr!FltReadFile+0x2ed [...]

A bit of reverse engineering offtopic here: I think that the absence of a trap means that cache processing NTFS code reuses page fault handling code.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Assembly Language, Crash Dump Analysis, Crash Dump Patterns, Virtualization, Windows Server 2008 | 2 Comments »

Crash Dump Analysis AntiPatterns (Part 13)

Thursday, July 9th, 2009

Sometimes we follow the course of established troubleshooting or debugging methods without questioning them or not considering possible exceptions. For example, the usual advice to heap corruption signs in process memory dumps is to ask to enable full page heap. However page heap helps to catch buffer overwrites but not underwrites and heap can also be damaged by other means like double free, passing an invalid address or direct corruption of control structures via a dangling pointer. Therefore in some cases when we enable full page heap we get the same stack trace:

Normal:

0:030> kL 100 ChildEBP RetAddr 175af77c 7d620ec5 ntdll_7d600000!RtlpCoalesceFreeBlocks+0×231 175af864 7c34218a ntdll_7d600000!RtlFreeHeap+0×38e 175af8ac 67d6153d msvcr71!free+0xc3 175af950 67781d69 Component!OnData+0×504 175af96c 7da4d03d Component!DispatchRequest+0×99 175afc28 7da4d177 rpcrt4!DispatchToStubInCNoAvrf+0×38 175afc7c 7da4d812 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f 175afca0 7da59a1c rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3 175afcfc 7da4da21 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×421 175afd20 7da3db14 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127 175aff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430 175aff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd 175affac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0×9d 175affb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0×1b 175affec 00000000 kernel32!BaseThreadStart+0×34

With full page heap enabled:

0:030> kL ChildEBP RetAddr 6092f568 7d6822fa ntdll!RtlpDphIsNormalHeapBlock+0×1f 6092f598 7d68256c ntdll!RtlpDphNormalHeapFree+0×21 6092f5f0 7d685443 ntdll!RtlpDebugPageHeapFree+0×146 6092f658 7d65714a ntdll!RtlDebugFreeHeap+0×2c 6092f730 7d62c5c0 ntdll!RtlFreeHeapSlowly+0×37 6092f814 7c34218a ntdll!RtlFreeHeap+0×11a 6092f85c 67d6153d msvcr71!free+0xc3 6092f900 67781d69 Component!OnData+0×504 6092f91c 7da4d03d Component!DispatchRequest+0×99 6092fc10 7da7ec5e rpcrt4!DispatchToStubInCNoAvrf+0×38 6092fc28 7da4d177 rpcrt4!DispatchToStubInCAvrf+0×14 6092fc7c 7da4d812 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f 6092fca0 7da59a1c rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3 6092fcfc 7da4da21 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×421 6092fd20 7da3db14 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127 6092ff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430 6092ff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd 6092ffac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0×9d 6092ffb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0×1b 6092ffec 00000000 kernel32!BaseThreadStart+0×34

We see that we get the same Component stack trace before calling free (shown in blue above) so we better search in a database of stack trace signatures before sending Habitual Reply to enable page heap. If there is a match in database then the course of future actions can be completely different: an immediate available hotfix or we find a similar problem that is already under investigation. The sooner we provide a relief to our customer the better.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in AntiPatterns, Crash Dump Analysis, Crash Dump Patterns | 5 Comments »

Virtualized process, incorrect stack trace, stack trace collection, multiple exceptions, optimized code and C++ exception: pattern cooperation

Wednesday, July 8th, 2009

It happened to me that I accidentally opened a YouTube page and it started playing video so I tried to either stop it or pushed the back browser button. Then immediately WER dialog appeared ruining my intricate multi-tab window. Nevertheless I looked at this event from the positive perspective as an opportunity to analyze a dump file and write a case study. I saved the crash dump using Windows Server 2008 Task Manager for later postmortem analysis.

Today I had a chance to look at it. When we open the dump file we see that it is the dump of a WOW64 process:

0:000> !analyze -v

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 0000000000000000 ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 0

STACK_TEXT: 00000000`0016ed78 00000000`75fcab46 : 00000000`77ddc990 00000000`0016fd20 00000000`75efa2b1 00000000`0016f760 : wow64cpu!WaitForMultipleObjects32+0×3a 00000000`0016ee20 00000000`75fca14c : 00000000`00000000 00000000`00000000 00000000`75fc3258 00000000`7ffe0030 : wow64!RunCpuSimulation+0xa 00000000`0016ee50 00000000`77d052d3 : 00000000`77cd0000 00000000`00000000 00000000`77de61b0 00000000`77cd0000 : wow64!Wow64LdrpInitialize+0×4b4 00000000`0016f3b0 00000000`77d05363 : 00000000`00000000 00000000`00000000 00000000`7efdf000 00000000`77d07ca7 : ntdll!LdrpInitializeProcess+0×14ac 00000000`0016f660 00000000`77cf85ce : 00000000`0016f760 00000000`00000000 00000000`7efdf000 00000000`00000000 : ntdll! ?? ::FNODOBFM::`string’+0×1ff19 00000000`0016f710 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

FOLLOWUP_IP: wow64cpu!WaitForMultipleObjects32+3a 00000000`75ec374f 418bbda0000000 mov edi,dword ptr [r13+0A0h]

MODULE_NAME: wow64cpu

Then we load a WOW64 extension that allows us to analyze 32-bit parts of saved 64-bit dumps and we retry the default analysis:

0:000> .load wow64exts; .effmach x86 Effective machine: x86 compatible (x86)

0:000:x86> !analyze -v

IP_ON_HEAP: ffffffffc0d84c02 The fault address in not in any loaded module, please check your build's rebase log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may contain the address if it were loaded.

STACK_TEXT: 00000000`003b0068 iexplore!__dyn_tls_init_callback <PERF> +0×0 ffffffff`c0d84c02 unknown+0×0 ffffffff`aacf27dc unknown+0×0 ffffffff`c1d84c02 unknown+0×0 ffffffff`b3cf38dc unknown+0×0 ffffffff`c2d84c02 unknown+0×0 00000000`3ecf38dc unknown+0×0 […] 00000000`7aca19dd unknown+0×0 ffffffff`e0d84c02 unknown+0×0 ffffffff`a7ca19dd unknown+0×0 ffffffff`

FOLLOWUP_IP: iexplore!__dyn_tls_init_callback <PERF> (iexplore+0x80068)+0 003b0068 70f7 jo iexplore!__dyn_tls_init_callback <PERF> (iexplore+0x80061) (003b0061)

Clearly the shown above stack trace is incorrect but at the same time the first thread stack looks consistent:

0:000:x86> k 100 ChildEBP RetAddr 0051d930 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15 0051d9cc 77888f76 kernel32!WaitForMultipleObjectsEx+0x11d 0051da20 7617ab28 user32!RealMsgWaitForMultipleObjectsEx+0x14d 0051da48 7617ac88 ole32!CCliModalLoop::BlockFn+0x97 0051da70 7617ad74 ole32!ModalLoop+0x5b 0051da80 76287c28 ole32!SwitchSTA+0x21 0051daac 762889d4 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x127 0051db8c 7617ad2e ole32!CRpcChannelBuffer::SendReceive2+0xef 0051dba8 7617ace0 ole32!CCliModalLoop::SendReceive+0x1e 0051dc20 7619e688 ole32!CAptRpcChnl::SendReceive+0x73 0051dc74 779f1074 ole32!CCtxComChnl::SendReceive+0x1c5 0051dc90 779f102b rpcrt4!NdrProxySendReceive+0x49 0051dc9c 779f0146 rpcrt4!NdrpProxySendReceive+0xb 0051e0a4 779f11ee rpcrt4!NdrClientCall2+0x18f 0051e0c8 779649c2 rpcrt4!ObjectStublessClient+0x90 0051e0d8 74bc86e3 rpcrt4!ObjectStubless+0xf WARNING: Frame IP not in any known module. Following frames may be wrong. 0051e0e4 74bda9bb ieframe+0x1986e3 0051e0f0 74b020d9 ieframe+0x1aa9bb 0051e164 74adbb6a ieframe+0xd20d9 0051e18c 77888807 ieframe+0xabb6a 0051e1b8 77888962 user32!InternalCallWinProc+0x23 0051e230 7788c4b6 user32!UserCallWinProcCheckWow+0x109 0051e274 7788c517 user32!SendMessageWorker+0x55b 0051e298 75b3f74a user32!SendMessageW+0x7f 0051e2c0 75b3f63b comctl32!CToolbar::TBOnLButtonUp+0x12f 0051e368 75b37020 comctl32!CToolbar::ToolbarWndProc+0xaa9 0051e388 77888807 comctl32!CToolbar::s_ToolbarWndProc+0x9b 0051e3b4 77888962 user32!InternalCallWinProc+0x23 0051e42c 77888aad user32!UserCallWinProcCheckWow+0x109 0051e490 77888b00 user32!DispatchMessageWorker+0x380 0051e4a0 74ae07ce user32!DispatchMessageW+0xf 0051e4d4 74a7ffce ieframe+0xb07ce 0051e4e0 74a6f579 ieframe+0x4ffce 0051e504 74a6f4c7 ieframe+0x3f579 0051f574 74a5d1ba ieframe+0x3f4c7 0051f7a4 003333c3 sxs!_hmod__RPCRT4_dll <PERF> (sxs+0x8d1ba) 0051fbe8 0033325a iexplore!wWinMain+0x27b 0051fc7c 775de4a5 iexplore!_initterm_e+0x1b1 0051fc88 77eecfed kernel32!BaseThreadInitThunk+0xe 0051fcc8 77eed1ff ntdll_77e70000!__RtlUserThreadStart+0x23 0051fce0 00000000 ntdll_77e70000!_RtlUserThreadStart+0x1b

We look at the stack trace collection to find another exception and we see it indeed on 7th stack trace:

0:000:x86> ~*k 100

[...]

7 Id: b44.b1c Suspend: 0 Teb: 7efad000 Unfrozen ChildEBP RetAddr 05baee9c 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15 05baef38 775649d9 kernel32!WaitForMultipleObjectsEx+0x11d 05baef54 7761573d kernel32!WaitForMultipleObjects+0x18 05baefc0 77615969 kernel32!WerpReportFaultInternal+0x16d 05baefd4 775ec66f kernel32!WerpReportFault+0×70 05baf060 77eed03e kernel32!UnhandledExceptionFilter+0×1b5 05baf068 77ebf2d0 ntdll_77e70000!__RtlUserThreadStart+0×6f 05baf07c 77f229b3 ntdll_77e70000!_EH4_CallFilterFunc+0×12 05baf0a4 77e93099 ntdll_77e70000!_except_handler4+0×8e 05baf0c8 77e9306b ntdll_77e70000!ExecuteHandler2+0×26 05baf178 77e92eff ntdll_77e70000!ExecuteHandler+0×24 05baf198 7757f328 ntdll_77e70000!KiUserExceptionDispatcher+0xf 05baf4fc 7155dead kernel32!RaiseException+0×58 WARNING: Stack unwind information not available. Following frames may be wrong. 05baf534 7155a59d Flash10b!DllUnregisterServer+0×1adbe0 05baf550 7150ca25 Flash10b!DllUnregisterServer+0×1aa2d0 05baf564 7150b5b5 Flash10b!DllUnregisterServer+0×15c758 05baf588 7150b90a Flash10b!DllUnregisterServer+0×15b2e8 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

[...]

97 Id: b44.ff4 Suspend: 1 Teb: 7ee9f000 Unfrozen ChildEBP RetAddr 7387facc 77561270 ntdll_77e70000!ZwWaitForSingleObject+0x15 7387fb3c 775611d8 kernel32!WaitForSingleObjectEx+0xbe 7387fb50 713e0a11 kernel32!WaitForSingleObject+0x12 WARNING: Stack unwind information not available. Following frames may be wrong. 7387fb74 713e0d34 Flash10b!DllUnregisterServer+0x30744 7387fb78 775de4a5 Flash10b!DllUnregisterServer+0x30a67 7387fb84 77eecfed kernel32!BaseThreadInitThunk+0xe 7387fbc4 77eed1ff ntdll_77e70000!__RtlUserThreadStart+0x23 7387fbdc 00000000 ntdll_77e70000!_RtlUserThreadStart+0x1b

Then we switch to this thread and try to find and set the new exception context:

0:000:x86> ~7s ntdll_77e70000!NtWaitForMultipleObjects+0x15: 77e90bc5 c21400 ret 14h

0:007:x86> kv ChildEBP RetAddr Args to Child 05baee9c 7756e91a 00000002 05baeeec 00000001 ntdll_77e70000!NtWaitForMultipleObjects+0x15 05baef38 775649d9 05baeeec 05baef88 00000000 kernel32!WaitForMultipleObjectsEx+0x11d 05baef54 7761573d 00000002 05baef88 00000000 kernel32!WaitForMultipleObjects+0x18 05baefc0 77615969 05baf090 00000001 00000001 kernel32!WerpReportFaultInternal+0x16d 05baefd4 775ec66f 05baf090 00000001 19e00de6 kernel32!WerpReportFault+0x70 05baf060 77eed03e 00000000 77ebf2d0 00000000 kernel32!UnhandledExceptionFilter+0×1b5 05baf068 77ebf2d0 00000000 05bafacc 77e9da38 ntdll_77e70000!__RtlUserThreadStart+0×6f 05baf07c 77f229b3 00000000 00000000 00000000 ntdll_77e70000!_EH4_CallFilterFunc+0×12 05baf0a4 77e93099 fffffffe 05bafabc 05baf1e0 ntdll_77e70000!_except_handler4+0×8e 05baf0c8 77e9306b 05baf190 05bafabc 05baf1e0 ntdll_77e70000!ExecuteHandler2+0×26 05baf178 77e92eff 01baf190 05baf1e0 05baf190 ntdll_77e70000!ExecuteHandler+0×24 05baf198 7757f328 00000003 19930520 05baf544 ntdll_77e70000!KiUserExceptionDispatcher+0xf 05baf4fc 7155dead e06d7363 00000001 00000003 kernel32!RaiseException+0×58 WARNING: Stack unwind information not available. Following frames may be wrong. 05baf534 7155a59d 05baf544 715e7954 715e5ac0 Flash10b!DllUnregisterServer+0×1adbe0 05baf550 7150ca25 002c1f7c 4237d4f8 4237c598 Flash10b!DllUnregisterServer+0×1aa2d0 05baf564 7150b5b5 002c1f60 00000010 423d0b28 Flash10b!DllUnregisterServer+0×15c758 05baf588 7150b90a 4237cb38 7150ba0b 00000000 Flash10b!DllUnregisterServer+0×15b2e8 00000000 00000000 00000000 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

Unfortunately a pointer to an exception pointers structure is NULL (probably because of optimized code) and we cannot use .exptr command. However we can still use a technique described in Hidden Exception pattern using 32-bit raw stack to set the context and display the exception record:

0:007:x86> !wow64exts.info

PEB32: 0x7efde000 PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0x7efaf000 TEB64: 0x7efad000

32 bit, StackBase : 0×5bb0000 StackLimit : 0×5b95000 Deallocation: 0×5ab0000

64 bit, StackBase : 0x30ffd20 StackLimit : 0x30f8000 Deallocation: 0x30c0000

0:007:x86> dps 0×5b95000 0×5bb0000 05b95000 00000000 05b95004 00000000 05b95008 00000000 […] 05baf160 00000009 05baf164 00000000 05baf168 00000000 05baf16c 05bb0000 05baf170 05b95000 05baf174 00000000 05baf178 05baf4fc 05baf17c 77e92eff ntdll_77e70000!KiUserExceptionDispatcher+0xf 05baf180 01baf190 05baf184 05baf1e0 ; .cxr 05baf188 05baf190 ; .exr 05baf18c 05baf1e0 05baf190 e06d7363 […]

0:007:x86> .cxr 05baf1e0 eax=05baf4ac ebx=002c1f60 ecx=00000003 edx=00000000 esi=716d9c54 edi=00000010 eip=7757f328 esp=05baf4ac ebp=05baf4fc iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206 kernel32!RaiseException+0×58: 7757f328 c9 leave

0:007:x86> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child 05baf4fc 7155dead e06d7363 00000001 00000003 kernel32!RaiseException+0×58 WARNING: Stack unwind information not available. Following frames may be wrong. 05baf534 7155a59d 05baf544 715e7954 715e5ac0 Flash10b!DllUnregisterServer+0×1adbe0 05baf550 7150ca25 002c1f7c 4237d4f8 4237c598 Flash10b!DllUnregisterServer+0×1aa2d0 05baf564 7150b5b5 002c1f60 00000010 423d0b28 Flash10b!DllUnregisterServer+0×15c758 05baf588 7150b90a 4237cb38 7150ba0b 00000000 Flash10b!DllUnregisterServer+0×15b2e8 00000000 00000000 00000000 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

0:007:x86> .exr 05baf190 ExceptionAddress: 000000007757f328 (kernel32!RaiseException+0×0000000000000058) ExceptionCode: e06d7363 (C++ EH exception) ExceptionFlags: 00000001 NumberParameters: 3 Parameter[0]: 0000000019930520 Parameter[1]: 0000000005baf544 Parameter[2]: 00000000715e7954

We double check that Flash10b code raised the exception indeed:

0:007:x86> .asm no_code_bytes Assembly options: no_code_bytes

0:007:x86> ub 7155dead Flash10b!DllUnregisterServer+0×1adbc4: 7155de91 je Flash10b!DllUnregisterServer+0×1adbcd (7155de9a) 7155de93 mov dword ptr [ebp-0Ch],1994000h 7155de9a lea eax,[ebp-0Ch] 7155de9d push eax 7155de9e push dword ptr [ebp-10h] 7155dea1 push dword ptr [ebp-1Ch] 7155dea4 push dword ptr [ebp-20h] 7155dea7 call dword ptr [Flash10b!DllUnregisterServer+0×1ba0a3 (7156a370)]

0:007:x86> dps 7156a370 7156a370 7757f2cf kernel32!RaiseException 7156a374 77562728 kernel32!WideCharToMultiByte 7156a378 7756c60b kernel32!GlobalFree 7156a37c 77562ddf kernel32!GetCurrentThread 7156a380 775739fe kernel32!SetThreadAffinityMask 7156a384 775654c9 kernel32!VirtualQuery 7156a388 77562e9c kernel32!IsDBCSLeadByte 7156a38c 77561ce5 kernel32!GetACP 7156a390 77565ed6 kernel32!GetCPInfo 7156a394 77561b36 kernel32!MultiByteToWideChar 7156a398 7756eb64 kernel32!ResetEvent 7156a39c 77562c88 kernel32!CreateEventA 7156a3a0 77561a3a kernel32!CloseHandle 7156a3a4 775649c1 kernel32!WaitForMultipleObjects 7156a3a8 77562c64 kernel32!SetEvent 7156a3ac 77560e18 kernel32!InterlockedExchange 7156a3b0 77560e2c kernel32!InterlockedCompareExchange 7156a3b4 77560c79 kernel32!Sleep 7156a3b8 77e935c0 ntdll_77e70000!RtlLeaveCriticalSection 7156a3bc 77e93580 ntdll_77e70000!RtlEnterCriticalSection 7156a3c0 77ea27fe ntdll_77e70000!RtlDeleteCriticalSection 7156a3c4 77ea2512 ntdll_77e70000!RtlInitializeCriticalSection 7156a3c8 77568c28 kernel32!WriteConsoleA 7156a3cc 7757a6dc kernel32!SetStdHandle 7156a3d0 77568c0b kernel32!GetConsoleMode 7156a3d4 77568c75 kernel32!GetConsoleCP 7156a3d8 77564ab7 kernel32!GetStringTypeW 7156a3dc 77576a56 kernel32!GetStringTypeA 7156a3e0 77562d90 kernel32!InitializeCriticalSectionAndSpinCount 7156a3e4 775608d0 kernel32!GetSystemTimeAsFileTime 7156a3e8 77565c3b kernel32!GetEnvironmentStringsW 7156a3ec 77565c13 kernel32!FreeEnvironmentStringsA

I believe the appearance of “dead” in the return address 7155dead was just a coincidence.

So we have found the 3rd-party component that raised e06d7363 C++ exception. Actually Internet search shows that e06d7363 is quite common in many applications.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Hunting for a Driver

Monday, July 6th, 2009

Consider this redirector file system exception while servicing a page fault:

1: kd> k 100 ChildEBP RetAddr afa8b2d4 f53442c4 nt!KeBugCheckEx+0x1b afa8b2fc f533df48 rdbss!RxExceptionFilter+0xf0 afa8b308 8083a93d rdbss!RxFsdCommonDispatch+0×3ca afa8b330 8083a498 nt!_except_handler3+0×61 afa8b354 8083a46a nt!ExecuteHandler2+0×26 afa8b404 80816479 nt!ExecuteHandler+0×24 afa8b7bc 808346c4 nt!KiDispatchException+0×131 afa8b824 80834678 nt!CommonDispatchException+0×4a afa8b848 808401e9 nt!KiExceptionExit+0×186 afa8b8ac f5345d44 nt!IopAllocateIrpPrivate+0xa3 afa8b8f8 f534619c rdbss!RxShadowIoHandler+0xac afa8b924 f534adb3 rdbss!RxShadowLowIo+0×1a1 afa8b94c f5350fa7 rdbss!RxLowIoSubmit+0×182 afa8b970 f5350dbd rdbss!RxLowIoReadShell+0×7d afa8ba1c f533d8d9 rdbss!RxCommonRead+0×416 afa8baac f534b9a2 rdbss!RxFsdCommonDispatch+0×320 afa8bacc f52c9a63 rdbss!RxFsdDispatch+0xd3 afa8baec 80840153 mrxsmb!MRxSmbFsdDispatch+0×134 afa8bb00 f71dcc45 nt!IofCallDriver+0×45 afa8bb28 80840153 fltmgr!FltpDispatch+0×6f afa8bb3c f421576a nt!IofCallDriver+0×45 afa8bb4c f4212621 DriverA!Dispatch+0xa4 afa8bb58 80840153 DriverA!Interface_dispatch+0×53 afa8bb6c f5251aef nt!IofCallDriver+0×45 afa8bb9c f5251bd2 DriverB!PassThrough+0×115 afa8bba8 80840153 DriverB!Dispatch+0×78 afa8bbbc f281ac29 nt!IofCallDriver+0×45 WARNING: Stack unwind information not available. Following frames may be wrong. afa8bbdc 8082784f DriverC+0×5c29 afa8bc14 80840153 nt!MiResolveMappedFileFault+0×640 afa8bc28 f780f42a nt!IofCallDriver+0×45 afa8bc40 80840153 DriverD+0×42a afa8bc54 80824b6f nt!IofCallDriver+0×45 afa8bc6c 8082645c nt!IoPageRead+0×109 afa8bcf0 80847650 nt!MiDispatchFault+0xd74 afa8bd4c 80836c2a nt!MmAccessFault+0×9c2 afa8bd4c 77df4749 nt!KiTrap0E+0xdc 0012f568 00000000 0×77df4749

We see that DriverA and DriverB are possibly pass-through and have little influence. For DriverC and DriverD we don’t even have symbol files but they are even further down the stack. Default analysis points to the code that tries to call another driver:

1: kd> !analyze -v [...] RDR_FILE_SYSTEM (27) If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the exception record and context record. Do a .cxr on the 3rd parameter and then kb to obtain a more informative stack trace. The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined as follows: RDBSS_BUG_CHECK_CACHESUP = 0xca550000, RDBSS_BUG_CHECK_CLEANUP = 0xc1ee0000, RDBSS_BUG_CHECK_CLOSE = 0xc10e0000, RDBSS_BUG_CHECK_NTEXCEPT = 0xbaad0000, Arguments: Arg1: baad0080 Arg2: afa8b7d8 Arg3: afa8b4d4 Arg4: 8084014f

EXCEPTION_RECORD: afa8b7d8 -- (.exr 0xffffffffafa8b7d8) ExceptionAddress: 8084014f (nt!IofCallDriver+0×00000041) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: 00000044 Attempt to read from address 00000044

CONTEXT: afa8b4d4 -- (.cxr 0xffffffffafa8b4d4) eax=00000003 ebx=00000000 ecx=86a62ed8 edx=861e0468 esi=00000000 edi=87083970 eip=8084014f esp=afa8b8a0 ebp=afa8b8ac iopl=0 nv up ei ng nz na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286 nt!IofCallDriver+0×41: 8084014f ff548638 call dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

STACK_TEXT: afa8b8ac f5345d44 86997a50 87083970 86863604 nt!IofCallDriver+0×41 afa8b8f8 f534619c 87083970 868634e0 e36c7ad8 rdbss!RxShadowIoHandler+0xac afa8b924 f534adb3 87083970 868634e0 e36c7ad8 rdbss!RxShadowLowIo+0×1a1 afa8b94c f5350fa7 01083970 868634e0 e36c7ad8 rdbss!RxLowIoSubmit+0×182 afa8b970 f5350dbd 87083970 868634e0 e36c7ad8 rdbss!RxLowIoReadShell+0×7d afa8ba1c f533d8d9 87083970 868634e0 88b56910 rdbss!RxCommonRead+0×416 afa8baac f534b9a2 f5348028 868634e0 88b56910 rdbss!RxFsdCommonDispatch+0×320 afa8bacc f52c9a63 88964768 868634e0 88d6c020 rdbss!RxFsdDispatch+0xd3 afa8baec 80840153 00000000 018634e0 88f372b8 mrxsmb!MRxSmbFsdDispatch+0×134 afa8bb00 f71dcc45 88f372b8 86863628 8843d418 nt!IofCallDriver+0×45 afa8bb28 80840153 88d6c020 868634e0 868634e0 fltmgr!FltpDispatch+0×6f afa8bb3c f421576a 00000000 88ad73f0 afa8bb6c nt!IofCallDriver+0×45 afa8bb4c f4212621 88d1ea98 868634e0 80840153 DriverA!Dispatch+0xa4 afa8bb58 80840153 88d1ea98 868634e0 88ada7f8 DriverA!Interface_dispatch+0×53 afa8bb6c f5251aef 86863604 88438a78 00000000 nt!IofCallDriver+0×45 afa8bb9c f5251bd2 88ada7f8 868634e0 80840153 DriverB!PassThrough+0×115 afa8bba8 80840153 88ada7f8 868634e0 88b3a0d8 DriverB!Dispatch+0×78 afa8bbbc f281ac29 00000000 87780ca0 afa8bc14 nt!IofCallDriver+0×45 WARNING: Stack unwind information not available. Following frames may be wrong. afa8bbdc 8082784f 88b3a0d8 81f2afa4 88ab83d8 DriverC+0×5c29 afa8bc14 80840153 88b3a020 868634e0 88433760 nt!MiResolveMappedFileFault+0×640 afa8bc28 f780f42a 8762cf38 f7811f2f 884336a8 nt!IofCallDriver+0×45 afa8bc40 80840153 884336a8 868634e0 868634e0 DriverD+0×42a afa8bc54 80824b6f 88ab83e8 86e0e438 88ab83d8 nt!IofCallDriver+0×45 afa8bc6c 8082645c 88b56906 88ab8410 88ab83f0 nt!IoPageRead+0×109 afa8bcf0 80847650 00000000 01fc897c c0007f20 nt!MiDispatchFault+0xd74 afa8bd4c 80836c2a 00000000 01fc897c 00000001 nt!MmAccessFault+0×9c2 afa8bd4c 77df4749 00000000 01fc897c 00000001 nt!KiTrap0E+0xdc 0012f568 00000000 00000000 00000000 00000000 0×77df4749

We assume that IofCallDriver has the same parameters as IoCallDriver but they were not passed on the stack. They are passed via ECX and EDX registers:

1: kd> !devobj 86997a50 86997a50: is not a device object

1: kd> !irp 87083970 IRP signature does not match, probably not an IRP

3: kd> .asm no_code_bytes Assembly options: no_code_bytes

1: kd> ub f5345d44 rdbss!RxShadowIoHandler+0×8d: f5345d25 mov dword ptr [ebp-4],ebx f5345d28 call dword ptr [rdbss!_imp__IoGetTopLevelIrp (f534711c)] f5345d2e mov dword ptr [ebp-20h],eax f5345d31 push ebx f5345d32 call dword ptr [rdbss!_imp__IoSetTopLevelIrp (f5347120)] f5345d38 mov edx,dword ptr [ebp-24h] f5345d3b mov ecx,dword ptr [ebp-28h] f5345d3e call dword ptr [rdbss!_imp_IofCallDriver (f5347204)]

We have the value of EBP at the time of the crash at IofCallDriver+0×41 so we need to see how it had changed:

1: kd> uf nt!IofCallDriver Flow analysis was incomplete, some code may be missing nt!IofCallDriver: 8084011b mov edi,edi 8084011d push ebp 8084011e mov ebp,esp 80840120 mov eax,dword ptr [nt!pIofCallDriver (808a5000)] 80840125 test eax,eax 80840127 jne nt!IofCallDriver+0xe (80859d6b)

nt!IofCallDriver+0x15: 8084012d dec byte ptr [edx+23h] 80840130 cmp byte ptr [edx+23h],0 80840134 jle nt!IofCallDriver+0x1e (80859d72)

nt!IofCallDriver+0x2c: 8084013a mov eax,dword ptr [edx+60h] 8084013d sub eax,24h 80840140 push esi 80840141 mov dword ptr [edx+60h],eax 80840144 mov dword ptr [eax+14h],ecx 80840147 movzx eax,byte ptr [eax] 8084014a mov esi,dword ptr [ecx+8] 8084014d push edx 8084014e push ecx 8084014f call dword ptr [esi+eax*4+38h] 80840153 pop esi 80840154 pop ebp 80840155 ret

nt!IofCallDriver+0xe: 80859d6b push dword ptr [ebp+4] 80859d6e call eax 80859d70 pop ebp 80859d71 ret

nt!IofCallDriver+0x1e: 80859d72 push 0 80859d74 push 0 80859d76 push 0 80859d78 push edx 80859d79 push 35h 80859d7b call nt!KeBugCheckEx (8087c485) 80859d80 int 3 80859d81 pop ebp 80859d82 jmp eax

We see the standard prolog and therefore EBP points to the old EBP from RxShadowIoHandler:

1: kd> dp ebp l1 afa8b8ac afa8b8f8

We have the value of the pointer to a device object, which is invalid:

1: kd> dp afa8b8f8-28 l1 afa8b8d0 86a62ed8

1: kd> !devobj 86a62ed8 86a62ed8: is not a device object

and the value of the pointer to an IRP which is valid:

1: kd> dp afa8b8f8-24 l1 afa8b8d4 861e0468

1: kd> !irp 861e0468 Irp is active with 4 stacks 4 is current (= 0x861e0544) Mdl=88ab8410: No System Buffer: Thread 8846e020: Irp stack trace. cmd flg cl Device File Completion-Context [ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000 >[ 3, 0] 0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel 86a62ed8: is not a device object rdbss!RxShadowIrpCompletion Args: 00008000 00000000 00008000 00000000

From the full disassembly code of IofCallDriver we see that ECX and EDX values had not been changed so we could get our parameters from them too.

Device object is invalid in IRP too and it points to a NULL driver object:

1: kd> dt _DEVICE_OBJECT 86a62ed8 nt!_DEVICE_OBJECT +0×000 Type : 0 +0×002 Size : 0 +0×004 ReferenceCount : 0 +0×008 DriverObject : (null) +0×00c NextDevice : (null) +0×010 AttachedDevice : (null) +0×014 CurrentIrp : (null) +0×018 Timer : (null) +0×01c Flags : 0 +0×020 Characteristics : 0 +0×024 Vpb : 0×00040000 _VPB +0×028 DeviceExtension : (null) +0×02c DeviceType : 0×86a62f04 +0×030 StackSize : 4 ” +0×034 Queue : __unnamed +0×05c AlignmentRequirement : 0 +0×060 DeviceQueue : _KDEVICE_QUEUE +0×074 Dpc : _KDPC +0×094 ActiveThreadCount : 0 +0×098 SecurityDescriptor : (null) +0×09c DeviceLock : _KEVENT +0×0ac SectorSize : 0 +0×0ae Spare1 : 0 +0×0b0 DeviceObjectExtension : 0×86a62f88 _DEVOBJ_EXTENSION +0×0b4 Reserved : 0×86a62f88

IofCallDriver tried to call a function pointer No.3 from the driver major dispatch table:

1: kd> dt _DRIVER_OBJECT nt!_DRIVER_OBJECT +0x000 Type : Int2B +0x002 Size : Int2B +0x004 DeviceObject : Ptr32 _DEVICE_OBJECT +0x008 Flags : Uint4B +0x00c DriverStart : Ptr32 Void +0x010 DriverSize : Uint4B +0x014 DriverSection : Ptr32 Void +0x018 DriverExtension : Ptr32 _DRIVER_EXTENSION +0x01c DriverName : _UNICODE_STRING +0x024 HardwareDatabase : Ptr32 _UNICODE_STRING +0x028 FastIoDispatch : Ptr32 _FAST_IO_DISPATCH +0x02c DriverInit : Ptr32 long +0x030 DriverStartIo : Ptr32 void +0x034 DriverUnload : Ptr32 void +0×038 MajorFunction : [28] Ptr32 long

The resulted effective address is a NULL code pointer (EAX=3, major code and ESI is NULL):

call dword ptr [esi+eax*4+38h] ds:0023:00000044=????????

Now we come back to our IRP to examine a file object:

cmd flg cl Device File Completion-Context >[ 3, 0] 0 e0 86a62ed8 86ff7028 f5345bdc-87083970 Success Error Cancel

1: kd> !fileobj 86ff7028

\Userdata\[…]\Application Suite\ApplicationSuite.RUS

Device Object: 0×87300aa8 \Driver\DriverE Vpb: 0×8843b280 Event signalled Access: Read SharedRead SharedDelete

Flags: 0x18c0040 Cache Supported Handle Created Fast IO Read Remote Origin

FsContext: 0xe453fda0 FsContext2: 0xe26904b8 CurrentByteOffset: 0 Cache Data: Section Object Pointers: 86b80a9c Shared Cache Map: 00000000

Here we see another DriverE and associated drivers from its device stack:

1: kd> !devstack 0×87300aa8 !DevObj !DrvObj !DevExt ObjectName 865f9cc8 \Driver\DriverF 865f9d80 8737aaf0 \Driver\DriverG 8737aba8 87155280 \Driver\DriverH 87155338 > 87300aa8 \Driver\DriverE 87300b60 BlockVolume1 !DevNode 88b6eac0 : DeviceInst is “STORAGE\Volume\{[…]}” ServiceName is “volsnap”

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Assembly Language, Crash Dump Analysis, Crash Dump Patterns | 1 Comment »

Blocked LPC thread, coupled processes, stack trace collection and blocked GUI thread: pattern cooperation

Saturday, July 4th, 2009

This small case study continues where Not using checklists common mistake case study left, after identifying the blocked LPC thread in ServiceA process. We know that ServiceA always asks (coupled with) ServiceB and, indeed, when looking at the latter stack trace collection we see a GUI thread showing a dialog box:

0:000> ~13kc

ntdll!KiFastSystemCallRet USER32!NtUserWaitMessage USER32!InternalDialogBox USER32!DialogBoxIndirectParamAorW USER32!DialogBoxIndirectParamW comdlg32!NewGetFileName comdlg32!NewGetSaveFileName comdlg32!GetFileName comdlg32!GetSaveFileNameW WARNING: Stack unwind information not available. Following frames may be wrong. DllA!DllEntryPoint ServiceB!Initialize […] ServiceB!ServiceThread kernel32!BaseThreadStart

From function names we can infer that the thread was displaying a “Save File As” common dialog box but the service was not allowed to interact with a desktop. This made the service blocked and, in return, made ServiceA blocked too. The suggestion was to enable ServiceB to interact with the desktop and keep an eye on DllA.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

10 Common Mistakes in Memory Analysis (Part 4)

Friday, July 3rd, 2009

One of the common mistakes that I observe is to habitually stick to certain WinDbg commands to recognize patterns. One example is !locks command used to find out any wait chains and deadlock conditions among threads. Recently a service process was reported to be hang and !locks command showed no blocked threads:

0:000> !locks CritSec +18caf94 at 018CAF94 LockCount -2 RecursionCount 1 OwningThread 58e8 EntryCount 0 ContentionCount 0 *** Locked

CritSec +18cc7c4 at 018CC7C4 LockCount -2 RecursionCount 1 OwningThread 58e8 EntryCount 0 ContentionCount 0 *** Locked

The number of threads waiting for the lock is 0 (this calculation is explained in the MSDN article):

0:000> ? ((-1) - (-2)) >> 2 Evaluate expression: 0 = 00000000

In the past, for that hang sevice memory dumps, !locks command always showed LockCount values corresponding to several waiting threads. Therefore, an engineer assumed that the dump was taken at some random time, not at the time the service was hanging, and asked for a new right dump. The mistake here is that the engineer didn’t look at the corresponding thread stack trace that shows the characteristic pattern of the blocked thread waiting for a reply from an LRPC call:

0:000> ~~[58e8]kc 100

ntdll!KiFastSystemCallRet ntdll!NtRequestWaitReplyPort RPCRT4!LRPC_CCALL::SendReceive RPCRT4!I_RpcSendReceive RPCRT4!NdrSendReceive RPCRT4!NdrClientCall2 ServiceA!foo […] ServiceA!bar RPCRT4!NdrStubCall2 RPCRT4!NdrServerCall2 RPCRT4!DispatchToStubInCNoAvrf RPCRT4!RPC_INTERFACE::DispatchToStubWorker RPCRT4!RPC_INTERFACE::DispatchToStub RPCRT4!RPC_INTERFACE::DispatchToStubWithObject RPCRT4!LRPC_SCALL::DealWithRequestMessage RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls RPCRT4!RecvLotsaCallsWrapper RPCRT4!BaseCachedThreadRoutine RPCRT4!ThreadStartRoutine kernel32!BaseThreadStart

We don’t see other blocked threads and wait chains because the dump was saved as soon as the freezing condition was detected: the service didn’t allow a user connection to proceed. If more users tried to connect we would have seen critical section wait chains that are absent in this dump.

To prevent such mistakes checklists are indispensable. For one example, see Crash Dump Analysis Checklist. You can also order it in print:

WinDbg: A Reference Poster and Learning Cards

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Common Mistakes, Crash Dump Analysis, Crash Dump Patterns, Debugging | 2 Comments »

Stack trace collection, message box, hidden exception, nested offender, insufficient memory, C++ exception, heap leak and ubiquitous component: pattern cooperation

Thursday, June 25th, 2009

My IE suddenly showed this message box:

I have never seen such message from IE so I rushed to Task Manager to save a dump. Default analysis (!analyze -v) was not able to find the problem:

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT

BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT

STACK_TEXT: 0031e624 77289244 7719c3e4 00000002 0031e678 ntdll!KiFastSystemCallRet 0031e628 7719c3e4 00000002 0031e678 00000001 ntdll!ZwWaitForMultipleObjects+0xc 0031e6c4 76f50208 0031e678 0031e6ec 00000000 kernel32!WaitForMultipleObjectsEx+0x11d 0031e718 70196071 00000028 0031e74c ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x13c 0031e738 701961f0 000004ff ffffffff 00000000 ieui!CoreSC::Wait+0x49 0031e760 70196196 000004ff 00000000 6f16074e ieui!CoreSC::WaitMessage+0x54 0031e76c 6f16074e 000c0ec8 00166038 00000000 ieui!WaitMessageEx+0x33 0031e79c 6f0fffce 00172e10 0031e7cc 6f0ef579 ieframe!CBrowserFrame::FrameMessagePump+0x199 0031e7a8 6f0ef579 00000000 00000000 000c0ec8 ieframe!BrowserThreadProc+0x3f 0031e7cc 6f0ef4c7 1ee1000a 000c0ec8 00000000 ieframe!BrowserNewThreadProc+0x7b 0031f83c 6f0dd1ba 000c0ec8 00000001 00000001 ieframe!SHOpenFolderWindow+0x188 0031fa6c 009033c3 000dd2c8 00000001 00910070 ieframe!IEWinMain+0x2d9 0031feb0 0090325a 00900000 00000000 000c1aa6 iexplore!wWinMain+0x27b 0031ff44 77194911 7ffd7000 0031ff90 7726e4b6 iexplore!_initterm_e+0x1b1 0031ff50 7726e4b6 7ffd7000 722730b1 00000000 kernel32!BaseThreadInitThunk+0xe 0031ff90 7726e489 009030dd 7ffd7000 00000000 ntdll!__RtlUserThreadStart+0x23 0031ffa8 00000000 009030dd 7ffd7000 00000000 ntdll!_RtlUserThreadStart+0x1b

However when browsing through stack trace collection I could spot a thread blocked by a message box, find another hidden exception and from it see the real nested offender that experienced insufficient memory condition resulted in C++ exception. You can see WinDbg output from this dump in the post about Nested Offender pattern (I don’t want to repeat it in this post).

Apart from that, the size of the memory dump, almost 1.8Gb, suggested a memory leak and we clearly see expanded heaps that also suggest the case of a heap leak:

0:087> !heap =s Index Address Name Debugging options enabled 1: 000c0000 Segment at 000c0000 to 001c0000 (00100000 bytes committed) Segment at 04510000 to 04610000 (00100000 bytes committed) Segment at 063b0000 to 065b0000 (00200000 bytes committed) Segment at 05f80000 to 06380000 (00400000 bytes committed) Segment at 0a8d0000 to 0b0d0000 (00800000 bytes committed) Segment at 0eab0000 to 0fa80000 (00fd0000 bytes committed) Segment at 160b0000 to 17080000 (00fd0000 bytes committed) Segment at 19020000 to 19ff0000 (00fd0000 bytes committed) Segment at 23fe0000 to 24fb0000 (00fd0000 bytes committed) Segment at 2ac20000 to 2bbf0000 (00fd0000 bytes committed) Segment at 34cc0000 to 35c90000 (00fd0000 bytes committed) Segment at 35fc0000 to 36f90000 (00fd0000 bytes committed) Segment at 40660000 to 41630000 (00fd0000 bytes committed) Segment at 45230000 to 46200000 (00fd0000 bytes committed) Segment at 4aed0000 to 4bea0000 (00fd0000 bytes committed) Segment at 4ee20000 to 4fdf0000 (00fd0000 bytes committed) Segment at 52eb0000 to 53e80000 (00fd0000 bytes committed) Segment at 53e80000 to 54e50000 (00fd0000 bytes committed) Segment at 575e0000 to 585b0000 (00fd0000 bytes committed) Segment at 58cb0000 to 59c80000 (00fd0000 bytes committed) Segment at 5ad00000 to 5bcd0000 (00fd0000 bytes committed) Segment at 5bcd0000 to 5cca0000 (00fd0000 bytes committed) Segment at 5ddb0000 to 5ed80000 (00fd0000 bytes committed) Segment at 77490000 to 78460000 (00fd0000 bytes committed) Segment at 78460000 to 79430000 (00fd0000 bytes committed) Segment at 7c420000 to 7d3f0000 (00fd0000 bytes committed) Segment at 7d3f0000 to 7e3c0000 (00fd0000 bytes committed) Segment at 7b690000 to 7be78000 (007e8000 bytes committed) Segment at 70470000 to 70864000 (003f4000 bytes committed) Segment at 72020000 to 72414000 (003f4000 bytes committed) Segment at 6ca70000 to 6cc6a000 (001fa000 bytes committed) Segment at 6d450000 to 6d64a000 (001fa000 bytes committed) Segment at 6c620000 to 6c81a000 (001fa000 bytes committed) Segment at 6e1b0000 to 6e3aa000 (001e6000 bytes committed) Segment at 701c0000 to 703ba000 (001ee000 bytes committed) Segment at 70ab0000 to 70caa000 (001ee000 bytes committed) Segment at 71770000 to 7196a000 (001e9000 bytes committed) Segment at 68060000 to 6825a000 (001f0000 bytes committed) Segment at 72a40000 to 72c3a000 (001ef000 bytes committed) Segment at 73170000 to 7336a000 (001f2000 bytes committed) Segment at 6d6c0000 to 6dab4000 (003b7000 bytes committed) Segment at 7a400000 to 7b3d0000 (00fb1000 bytes committed) Segment at 3c480000 to 3c57d000 (000d2000 bytes committed) Segment at 2e950000 to 2ea4d000 (000cc000 bytes committed) Segment at 7b3d0000 to 7b5ca000 (001c1000 bytes committed) Segment at 2ec60000 to 2ed5d000 (000c1000 bytes committed) Segment at 31570000 to 3166d000 (000c1000 bytes committed) Segment at 43050000 to 4314d000 (000c1000 bytes committed) Segment at 48f30000 to 4902d000 (000c1000 bytes committed) Segment at 492b0000 to 493ad000 (000c1000 bytes committed) Segment at 49bb0000 to 49cad000 (000c1000 bytes committed) Segment at 49d50000 to 49e4d000 (000c1000 bytes committed) Segment at 4bea0000 to 4bf9d000 (000c1000 bytes committed) Segment at 4d140000 to 4d23d000 (000c1000 bytes committed) Segment at 55040000 to 5513d000 (000c1000 bytes committed) Segment at 55180000 to 5527d000 (000c1000 bytes committed) Segment at 555c0000 to 556bd000 (000c1000 bytes committed) Segment at 557d0000 to 558cd000 (000c1000 bytes committed) Segment at 5a380000 to 5a47d000 (000c1000 bytes committed) Segment at 5a980000 to 5aa7d000 (000c1000 bytes committed) Segment at 5ab40000 to 5ac3d000 (000c1000 bytes committed) Segment at 6dce0000 to 6dddd000 (000c6000 bytes committed) Segment at 75680000 to 7577d000 (000c6000 bytes committed) Segment at 4d6f0000 to 4d7ed000 (000c1000 bytes committed) Segment at 2ca40000 to 2cabf000 (00041000 bytes committed) Segment at 4aa30000 to 4aaaf000 (00041000 bytes committed) Segment at 67c20000 to 67d1d000 (000c1000 bytes committed) Segment at 2e820000 to 2e91d000 (000c1000 bytes committed) Segment at 4e680000 to 4e77d000 (000c1000 bytes committed) Segment at 5d4c0000 to 5d5bd000 (000c1000 bytes committed) Segment at 683a0000 to 6849d000 (000c1000 bytes committed) Segment at 6a440000 to 6a53d000 (000c1000 bytes committed) Segment at 4d7f0000 to 4d86f000 (00041000 bytes committed) Segment at 60380000 to 603ff000 (00041000 bytes committed) Segment at 65460000 to 654df000 (00041000 bytes committed) Segment at 67fb0000 to 6802f000 (00041000 bytes committed) Segment at 684a0000 to 6851f000 (00041000 bytes committed) Segment at 6a540000 to 6a5bf000 (00041000 bytes committed) Segment at 6ab80000 to 6abff000 (00041000 bytes committed) Segment at 6b1e0000 to 6b25f000 (00041000 bytes committed) Segment at 6c390000 to 6c40f000 (00041000 bytes committed) Segment at 6dee0000 to 6df5f000 (00041000 bytes committed) Segment at 6e8d0000 to 6e94f000 (00041000 bytes committed) Segment at 6ef90000 to 6f00f000 (00041000 bytes committed) Segment at 72420000 to 7249f000 (00041000 bytes committed) Segment at 740d0000 to 7414f000 (00041000 bytes committed) Segment at 74c30000 to 74caf000 (00041000 bytes committed) Segment at 75150000 to 751cf000 (00041000 bytes committed) Segment at 7b5d0000 to 7b64f000 (00041000 bytes committed) Segment at 09d10000 to 09d51000 (00041000 bytes committed) Segment at 0ddb0000 to 0ddf1000 (00041000 bytes committed) Segment at 0e810000 to 0e851000 (00041000 bytes committed) Segment at 2c580000 to 2c5c1000 (00041000 bytes committed) Segment at 2d490000 to 2d4d1000 (00041000 bytes committed) Segment at 2ea50000 to 2ea91000 (00041000 bytes committed) Segment at 31060000 to 310a1000 (00041000 bytes committed) Segment at 322a0000 to 322e1000 (00041000 bytes committed) Segment at 323a0000 to 323e1000 (00041000 bytes committed) Segment at 32ff0000 to 33031000 (00041000 bytes committed) Segment at 33d90000 to 33dd1000 (00041000 bytes committed) Segment at 34330000 to 34371000 (00041000 bytes committed) Segment at 35c90000 to 35cd1000 (00041000 bytes committed) Segment at 37250000 to 37291000 (00041000 bytes committed) Segment at 40290000 to 402d1000 (00041000 bytes committed) Segment at 42ce0000 to 42d21000 (00041000 bytes committed) Segment at 42ff0000 to 43031000 (00041000 bytes committed) Segment at 48a40000 to 48a81000 (00041000 bytes committed) Segment at 55470000 to 554b1000 (00041000 bytes committed) Segment at 55930000 to 55971000 (00041000 bytes committed) Segment at 55a80000 to 55ac1000 (00041000 bytes committed) Segment at 59e90000 to 59ed1000 (00041000 bytes committed) Segment at 59ff0000 to 5a031000 (00041000 bytes committed) Segment at 5d730000 to 5d771000 (00041000 bytes committed) Segment at 5d810000 to 5d851000 (00041000 bytes committed) Segment at 60230000 to 60271000 (00041000 bytes committed) Segment at 6a5c0000 to 6a601000 (00041000 bytes committed) Segment at 6c5d0000 to 6c611000 (00041000 bytes committed) Segment at 6c8e0000 to 6c921000 (00041000 bytes committed) Segment at 6cfb0000 to 6cff1000 (00041000 bytes committed) Segment at 6dfc0000 to 6e001000 (00041000 bytes committed) Segment at 6e730000 to 6e771000 (00041000 bytes committed) Segment at 6eec0000 to 6ef01000 (00041000 bytes committed) Segment at 700e0000 to 70121000 (00041000 bytes committed) Segment at 703c0000 to 70401000 (00041000 bytes committed) Segment at 70870000 to 708b1000 (00041000 bytes committed) Segment at 70a40000 to 70a81000 (00041000 bytes committed) Segment at 71ca0000 to 71ce1000 (00041000 bytes committed) Segment at 729e0000 to 72a21000 (00041000 bytes committed) Segment at 73070000 to 730b1000 (00041000 bytes committed) Segment at 73410000 to 73451000 (00041000 bytes committed) Segment at 74a80000 to 74ac1000 (00041000 bytes committed) Segment at 75880000 to 758c1000 (00041000 bytes committed) Segment at 7f690000 to 7f6d1000 (00041000 bytes committed) Segment at 7fd70000 to 7fdb1000 (00041000 bytes committed) 2: 00010000 Segment at 00010000 to 00020000 (00003000 bytes committed) 3: 000a0000 Segment at 000a0000 to 000b0000 (00010000 bytes committed) Segment at 006a0000 to 007a0000 (00014000 bytes committed) 4: 00020000 Segment at 00020000 to 00030000 (00010000 bytes committed) Segment at 01780000 to 01880000 (00100000 bytes committed) Segment at 090b0000 to 092b0000 (00200000 bytes committed) Segment at 0cff0000 to 0d3f0000 (00400000 bytes committed) Segment at 0d3f0000 to 0dbf0000 (00800000 bytes committed) Segment at 10790000 to 11760000 (00fd0000 bytes committed) Segment at 143b0000 to 15380000 (00fd0000 bytes committed) Segment at 17080000 to 18050000 (00fd0000 bytes committed) Segment at 18050000 to 19020000 (00fd0000 bytes committed) Segment at 19ff0000 to 1afc0000 (00fd0000 bytes committed) Segment at 20be0000 to 21bb0000 (00fd0000 bytes committed) Segment at 24fb0000 to 25f80000 (00fd0000 bytes committed) Segment at 25f80000 to 26f50000 (00fd0000 bytes committed) Segment at 2d6b0000 to 2e680000 (00fd0000 bytes committed) Segment at 3a9c0000 to 3b990000 (00fd0000 bytes committed) Segment at 3c9c0000 to 3d990000 (00fd0000 bytes committed) Segment at 41a50000 to 42a20000 (00fd0000 bytes committed) Segment at 44260000 to 45230000 (00fd0000 bytes committed) Segment at 46200000 to 471d0000 (00fd0000 bytes committed) Segment at 471d0000 to 481a0000 (00fd0000 bytes committed) Segment at 4c170000 to 4d140000 (00fd0000 bytes committed) Segment at 4fdf0000 to 50dc0000 (00fd0000 bytes committed) Segment at 51ee0000 to 52eb0000 (00fd0000 bytes committed) Segment at 56400000 to 573d0000 (00fd0000 bytes committed) Segment at 68de0000 to 69db0000 (00fd0000 bytes committed) Segment at 79430000 to 7a400000 (00fd0000 bytes committed) Segment at 7e3c0000 to 7f390000 (00fd0000 bytes committed) Segment at 73820000 to 74008000 (007e8000 bytes committed) Segment at 6d030000 to 6d424000 (003f4000 bytes committed) Segment at 6f680000 to 6fa74000 (003f4000 bytes committed) Segment at 6eac0000 to 6eeb4000 (003f4000 bytes committed) Segment at 74210000 to 74604000 (003f4000 bytes committed) Segment at 7be80000 to 7c274000 (003f4000 bytes committed) Segment at 7f7f0000 to 7fbe4000 (003f4000 bytes committed) Segment at 6dae0000 to 6dcda000 (001fa000 bytes committed) Segment at 6fa80000 to 6fc7a000 (001fa000 bytes committed) Segment at 71440000 to 7163a000 (001fa000 bytes committed) Segment at 727e0000 to 729da000 (001fa000 bytes committed) Segment at 60450000 to 6064a000 (001fa000 bytes committed) Segment at 72e50000 to 7304a000 (001fa000 bytes committed) Segment at 5d070000 to 5d16d000 (000fd000 bytes committed) Segment at 5ac80000 to 5acff000 (00075000 bytes committed) Segment at 5d970000 to 5d9ef000 (0007f000 bytes committed) Segment at 5dc10000 to 5dc8f000 (0007f000 bytes committed) Segment at 5f0e0000 to 5f15f000 (0007f000 bytes committed) Segment at 6b0e0000 to 6b1dd000 (000dd000 bytes committed) Segment at 494c0000 to 496ba000 (001f2000 bytes committed) Segment at 6a240000 to 6a43a000 (001ec000 bytes committed) Segment at 376b0000 to 3772f000 (0007f000 bytes committed) 5: 008e0000 Segment at 008e0000 to 008f0000 (00010000 bytes committed) Segment at 007a0000 to 008a0000 (00100000 bytes committed) Segment at 5a6e0000 to 5a8e0000 (00041000 bytes committed) 6: 00210000 Segment at 00210000 to 00220000 (00010000 bytes committed) Segment at 050f0000 to 051f0000 (00022000 bytes committed) 7: 00080000 Segment at 00080000 to 00090000 (00010000 bytes committed) Segment at 05d80000 to 05e80000 (00022000 bytes committed) 8: 018f0000 Segment at 018f0000 to 01900000 (00010000 bytes committed) Segment at 041e0000 to 042e0000 (000fa000 bytes committed) Segment at 5eee0000 to 5f0e0000 (00066000 bytes committed) 9: 001c0000 Segment at 001c0000 to 001d0000 (00001000 bytes committed) 10: 018c0000 Segment at 018c0000 to 018d0000 (00003000 bytes committed) 11: 01dc0000 Segment at 01dc0000 to 01e00000 (00032000 bytes committed) 12: 037c0000 Segment at 037c0000 to 03800000 (0000a000 bytes committed) 13: 008c0000 Segment at 008c0000 to 008d0000 (00006000 bytes committed) Segment at 1fc90000 to 1fd90000 (00012000 bytes committed) 14: 03750000 Segment at 03750000 to 03760000 (00003000 bytes committed) 15: 03b20000 Segment at 03b20000 to 03b60000 (00016000 bytes committed) 16: 03a40000 Segment at 03a40000 to 03a80000 (00040000 bytes committed) Segment at 0a420000 to 0a520000 (000c6000 bytes committed) 17: 04050000 Segment at 04050000 to 04090000 (00040000 bytes committed) 18: 04340000 Segment at 04340000 to 04380000 (00040000 bytes committed) 19: 04500000 Segment at 04500000 to 04510000 (0000e000 bytes committed) Segment at 08b30000 to 08c30000 (000e9000 bytes committed) Segment at 2cfe0000 to 2d0e0000 (00022000 bytes committed) 20: 04800000 Segment at 04800000 to 04900000 (00100000 bytes committed) 21: 049c0000 Segment at 049c0000 to 049d0000 (00010000 bytes committed) Segment at 049d0000 to 04ad0000 (00023000 bytes committed) 22: 04c50000 Segment at 04c50000 to 04c60000 (00010000 bytes committed) Segment at 04ad0000 to 04bd0000 (00100000 bytes committed) Segment at 1fa90000 to 1fc90000 (00200000 bytes committed) Segment at 49710000 to 49b10000 (003d1000 bytes committed) 23: 05200000 Segment at 05200000 to 05300000 (00100000 bytes committed) 24: 053e0000 Segment at 053e0000 to 05420000 (0001e000 bytes committed) 25: 051f0000 Segment at 051f0000 to 05200000 (00010000 bytes committed) Segment at 0a100000 to 0a200000 (00022000 bytes committed) 26: 040d0000 Segment at 040d0000 to 040e0000 (00010000 bytes committed) Segment at 05570000 to 05670000 (00012000 bytes committed) 27: 047e0000 Segment at 047e0000 to 047f0000 (00010000 bytes committed) Segment at 05e80000 to 05f80000 (00026000 bytes committed) 28: 05560000 Segment at 05560000 to 05570000 (00010000 bytes committed) Segment at 05420000 to 05520000 (00032000 bytes committed) 29: 06b30000 Segment at 06b30000 to 06b40000 (00010000 bytes committed) Segment at 1ff30000 to 20030000 (00080000 bytes committed) 30: 053b0000 Segment at 053b0000 to 053c0000 (00010000 bytes committed) Segment at 08df0000 to 08ef0000 (00100000 bytes committed) Segment at 2bbf0000 to 2bdf0000 (00200000 bytes committed) Segment at 316a0000 to 31aa0000 (003f9000 bytes committed) Segment at 31aa0000 to 322a0000 (0049c000 bytes committed) 31: 09460000 Segment at 09460000 to 094a0000 (0003e000 bytes committed) Segment at 14070000 to 14170000 (00100000 bytes committed) Segment at 51ce0000 to 51ee0000 (00124000 bytes committed) 32: 08cf0000 Segment at 08cf0000 to 08d30000 (00001000 bytes committed) 33: 09360000 Segment at 09360000 to 093a0000 (00001000 bytes committed) 34: 04f40000 Segment at 04f40000 to 04f80000 (00001000 bytes committed) 35: 09560000 Segment at 09560000 to 095a0000 (00001000 bytes committed) 36: 0b6e0000 Segment at 0b6e0000 to 0b6f0000 (00010000 bytes committed) Segment at 07720000 to 07820000 (000ca000 bytes committed) 37: 2c410000 Segment at 2c410000 to 2c420000 (00008000 bytes committed) 38: 29420000 Segment at 29420000 to 29460000 (00040000 bytes committed) 39: 3c120000 Segment at 3c120000 to 3c130000 (00002000 bytes committed) 40: 60410000 Segment at 60410000 to 60450000 (00040000 bytes committed) 41: 65500000 Segment at 65500000 to 65510000 (00006000 bytes committed) 42: 67d60000 Segment at 67d60000 to 67da0000 (00001000 bytes committed)

I actually wanted to run !heap -s command to get aggregated stats but misprint =s gave me the old format output you can see above that I enjoyed using previous versions of OS and WinDbg. Stats for 2 process heaps:

0:087> !heap -s -h 00020000

[...]

0: Heap 00020000 Flags 00001002 - HEAP_GROWABLE Reserved memory in segments 424788 (k) Commited memory in segments 424532 (k) Virtual bytes (correction for large UCR) 424788 (k) Free space 3298 (k) (1858 blocks) External fragmentation 0% (1858 free blocks) Virtual address fragmentation 0% (49 uncommited ranges) Virtual blocks 65 - total 0 KBytes Lock contention 18 Segments 1

Low fragmentation heap 01780048 Lock contention 0 Metadata usage 195584 bytes Statistics: Segments created 831950 Segments deleted 826741 Segments reused 0

[...]

0:087> !heap -s -h 000c0000

[...]

Walking the heap 000c0000 .. 0: Heap 000c0000 Flags 00000002 - HEAP_GROWABLE Reserved memory in segments 479980 (k) Commited memory in segments 467984 (k) Virtual bytes (correction for large UCR) 479980 (k) Free space 4273 (k) (2000 blocks) External fragmentation 0% (2000 free blocks) Virtual address fragmentation 2% (134 uncommited ranges) Virtual blocks 25 - total 0 KBytes Lock contention 9092 Segments 1

Low fragmentation heap 000c6a98 Lock contention 0 Metadata usage 326656 bytes Statistics: Segments created 1534237 Segments deleted 1524948 Segments reused 0

[...]

Instead of enabling user mode stack trace database or collecting UMDH logs I conjectured a weak link with the found ubiquitous module in the stack trace collection. You can find the discussion and corresponding WinDbg output from the same dump in Ubiquitous Component pattern description. After component removal the problem disappeared. Nice troubleshooting in one iteration.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | 2 Comments »

Crash Dump Analysis Patterns (Part 86)

Wednesday, June 24th, 2009

Sometimes we suspect one component but there is another, Nested Offender, that raised an exception. That exception propagated through exception filters and handlers and prompted the suspected component to respond with a diagnostic message. Here is an example of the thread showing the runtime error message box:

The corresponding stack trace points to MathMLMimer module:

0:087> kL ChildEBP RetAddr 5546ddf0 76f50dde ntdll!KiFastSystemCallRet 5546ddf4 76f3b0b2 user32!NtUserWaitMessage+0xc 5546de28 76f3bcda user32!DialogBox2+0x202 5546de50 76f8ccdc user32!InternalDialogBox+0xd0 5546def0 76f8d25e user32!SoftModalMessageBox+0x69f 5546e040 76f8d394 user32!MessageBoxWorker+0x2c7 5546e098 76f8d43e user32!MessageBoxTimeoutW+0x7f 5546e0cc 76f8d5ec user32!MessageBoxTimeoutA+0xa1 5546e0ec 6f245ac7 user32!MessageBoxExA+0x1b 5546e10c 76f8d65e ieframe!Detour_MessageBoxExA+0x2c 5546e128 0c841c28 user32!MessageBoxA+0×45 WARNING: Stack unwind information not available. Following frames may be wrong. 5546e16c 0c83d1b9 MathMLMimer!DllUnregisterServer+0×9437 5546e190 0c83cf72 MathMLMimer!DllUnregisterServer+0×49c8 00000000 00000000 MathMLMimer!DllUnregisterServer+0×4781

0:087> .asm no_code_bytes Assembly options: no_code_bytes

0:087> ub 0c841c28 MathMLMimer!DllUnregisterServer+0×941d: 0c841c0e push dword ptr [ebp+10h] 0c841c11 push dword ptr [ebp+0Ch] 0c841c14 push dword ptr [ebp+8] 0c841c17 push dword ptr [ebp-4] 0c841c1a push dword ptr [MathMLMimer!DllUnregisterServer+0×135ab (0c84bd9c)] 0c841c20 call MathMLMimer!DllUnregisterServer+0×4aab (0c83d29c) 0c841c25 pop ecx 0c841c26 call eax

However, when looking at raw stack data, we see exception processing residue that points to ComponentA that tried to allocate more memory than was available (bad_alloc C++ exception):

0:087> !teb TEB at 7ff84000 ExceptionList: 5546e4e8 StackBase: 55470000 StackLimit: 55457000 SubSystemTib: 00000000 FiberData: 00001e00 ArbitraryUserPointer: 00000000 Self: 7ff84000 EnvironmentPointer: 00000000 ClientId: 0000136c . 00001714 RpcHandle: 00000000 Tls Storage: 5826eef0 PEB Address: 7ffd7000 LastErrorValue: 0 LastStatusValue: 0 Count Owned Locks: 0 HardErrorMode: 0

0:087> dds 55457000 55470000 55457000 00000000 [...] 5546e694 55470000 5546e698 55457000 5546e69c 00274cb0 5546e6a0 5546e9f4 5546e6a4 772899f7 ntdll!KiUserExceptionDispatcher+0xf 5546e6a8 0046e6b8 5546e6ac 5546e6d8 5546e6b0 5546e6b8 […]

0:087> .cxr 5546e6d8 eax=5546e9a4 ebx=00001000 ecx=00000003 edx=00000000 esi=689a9c54 edi=6b330000 eip=771942eb esp=5546e9a4 ebp=5546e9f4 iopl=0 nv up ei pl nz ac po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00240212 kernel32!RaiseException+0×58: 771942eb c9 leave

0:087> .exr 5546e6b8 ExceptionAddress: 771942eb (kernel32!RaiseException+0×00000058) ExceptionCode: e06d7363 (C++ EH exception) ExceptionFlags: 00000001 NumberParameters: 3 Parameter[0]: 19930520 Parameter[1]: 5546ea3c Parameter[2]: 688b7954 pExceptionObject: 5546ea3c _s_ThrowInfo : 688b7954 Type : class std::bad_alloc Type : class std::exception

0:087> kL *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr 5546e9f4 6882dead kernel32!RaiseException+0×58 WARNING: Stack unwind information not available. Following frames may be wrong. 5546ea2c 6882a59d ComponentA!DllUnregisterServer+0×1adbe0 5546ea48 6868157b ComponentA!DllUnregisterServer+0×1aa2d0 5546ea74 6869d2c6 ComponentA!DllUnregisterServer+0×12ae 5546ea88 6868a415 ComponentA!DllUnregisterServer+0×1cff9 5546eaa0 685a165e ComponentA!DllUnregisterServer+0xa148 5546eac8 685a9828 ComponentA+0×5165e 5546ebc8 68605fdd ComponentA+0×59828 5546ebf0 6a807aba ComponentA+0xb5fdd 5546ec08 6a807a77 mshtml!CViewDispClient::Invalidate+0×59 5546ec20 00000000 mshtml!CDispRoot::InvalidateRoot+0×1d

0:087> ub 6882dead ComponentA!DllUnregisterServer+0×1adbc4: 6882de91 je ComponentA!DllUnregisterServer+0×1adbcd (6882de9a) 6882de93 mov dword ptr [ebp-0Ch],1994000h 6882de9a lea eax,[ebp-0Ch] 6882de9d push eax 6882de9e push dword ptr [ebp-10h] 6882dea1 push dword ptr [ebp-1Ch] 6882dea4 push dword ptr [ebp-20h] 6882dea7 call dword ptr [ComponentA!DllUnregisterServer+0×1ba0a3 (6883a370)]

It happens that MathMLMimer has an exception filter that shows the runtime error:

0:087> !exchain 5546e4e8: MathMLMimer!DllUnregisterServer+18df (0c83a0d0) 5546e578: kernel32!_except_handler4+0 (7715e289) 5546e5e4: ntdll!ExecuteHandler2+3a (77289bad) 5546faa8: user32!_except_handler4+0 (76f951ba) 5546fb0c: user32!_except_handler4+0 (76f951ba) 5546fbd0: ntdll!_except_handler4+0 (77239834) Invalid exception stack at ffffffff

Notice the mystical 6882dead return address above. This is just a coincidence, I believe.

Nested Offender is different from Nested Exception pattern. The latter is about an exception handler that experiences or throws another exception.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | 2 Comments »

Crash Dump Analysis Patterns (Part 85)

Tuesday, June 23rd, 2009

Sometimes we look at a stack trace collection and we see dozens of threads running through some 3rd-party component. We do not normally expect this component to appear or expect only one or two threads running through it. Here is an example from my IE after installing a 3rd-party toolbar that becomes Ubiquitous Component (the component has been renamed not “to awaken a sleeping giant and fill him with a terrible resolve”):

0:087> ~*kc

# 0 Id: 136c.fa0 Suspend: 0 Teb: 7ffdf000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx user32!RealMsgWaitForMultipleObjectsEx ieui!CoreSC::Wait ieui!CoreSC::WaitMessage ieui!WaitMessageEx ieframe!CBrowserFrame::FrameMessagePump ieframe!BrowserThreadProc ieframe!BrowserNewThreadProc ieframe!SHOpenFolderWindow ieframe!IEWinMain iexplore!wWinMain iexplore!_initterm_e kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

1 Id: 136c.12c0 Suspend: 0 Teb: 7ffdc000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects ntdll!TppWaiterpThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

2 Id: 136c.1120 Suspend: 0 Teb: 7ffda000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx user32!RealMsgWaitForMultipleObjectsEx ieui!CoreSC::Wait ieui!CoreSC::xwProcessNL ieui!GetMessageExA ieui!ResourceManager::SharedThreadProc msvcrt!_endthreadex msvcrt!_endthreadex kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

3 Id: 136c.1588 Suspend: 0 Teb: 7ffd9000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

4 Id: 136c.15a4 Suspend: 0 Teb: 7ffd8000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

5 Id: 136c.111c Suspend: 0 Teb: 7ffd6000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForWorkViaWorkerFactory ntdll!TppWorkerThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

6 Id: 136c.10a8 Suspend: 0 Teb: 7ffd5000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForSingleObject kernel32!WaitForSingleObjectEx kernel32!WaitForSingleObject WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

7 Id: 136c.fb0 Suspend: 0 Teb: 7ffd3000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

8 Id: 136c.1728 Suspend: 0 Teb: 7ffae000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

9 Id: 136c.918 Suspend: 0 Teb: 7ffad000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

10 Id: 136c.11c4 Suspend: 0 Teb: 7ffd4000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForSingleObject mswsock!SockWaitForSingleObject mswsock!WSPSelect ws2_32!select wininet!ICAsyncThread::SelectThread wininet!ICAsyncThread::SelectThreadWrapper kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

11 Id: 136c.13bc Suspend: 0 Teb: 7ffa8000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForSingleObject kernel32!WaitForSingleObjectEx kernel32!WaitForSingleObject WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

12 Id: 136c.178 Suspend: 0 Teb: 7ffab000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserGetMessage user32!GetMessageA winmm!mciwindow kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

13 Id: 136c.1594 Suspend: 0 Teb: 7ffa9000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx wdmaud!CWorker::_ThreadProc wdmaud!CWorker::_StaticThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

14 Id: 136c.b50 Suspend: 0 Teb: 7ffa0000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

15 Id: 136c.eec Suspend: 0 Teb: 7ff9e000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

16 Id: 136c.664 Suspend: 0 Teb: 7ff9d000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

17 Id: 136c.2cc Suspend: 0 Teb: 7ff9b000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

18 Id: 136c.e00 Suspend: 0 Teb: 7ff9c000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForSingleObject kernel32!WaitForSingleObjectEx kernel32!WaitForSingleObject mshtml!CTimerMan::ThreadExec mshtml!CExecFT::ThreadProc mshtml!CExecFT::StaticThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

19 Id: 136c.620 Suspend: 0 Teb: 7ff95000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

20 Id: 136c.1158 Suspend: 0 Teb: 7ff91000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

21 Id: 136c.10cc Suspend: 0 Teb: 7ff90000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

22 Id: 136c.1264 Suspend: 0 Teb: 7ff8f000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

23 Id: 136c.13fc Suspend: 0 Teb: 7ffde000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

24 Id: 136c.914 Suspend: 0 Teb: 7ffdb000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

25 Id: 136c.1194 Suspend: 0 Teb: 7ffac000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

26 Id: 136c.1548 Suspend: 0 Teb: 7ffa6000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

27 Id: 136c.a00 Suspend: 0 Teb: 7ff99000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

28 Id: 136c.1360 Suspend: 0 Teb: 7ff97000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

29 Id: 136c.dc0 Suspend: 0 Teb: 7ff96000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

30 Id: 136c.a80 Suspend: 0 Teb: 7ff94000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

31 Id: 136c.1390 Suspend: 0 Teb: 7ffa1000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForSingleObject kernel32!WaitForSingleObjectEx kernel32!WaitForSingleObject WARNING: Stack unwind information not available. Following frames may be wrong. Flash10b!DllUnregisterServer ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

32 Id: 136c.123c Suspend: 0 Teb: 7ff9f000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForSingleObject kernel32!WaitForSingleObjectEx kernel32!WaitForSingleObject WARNING: Stack unwind information not available. Following frames may be wrong. Flash10b!DllUnregisterServer ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

33 Id: 136c.1398 Suspend: 0 Teb: 7ff82000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects winmm!timeThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

34 Id: 136c.ca0 Suspend: 0 Teb: 7ff7d000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx user32!RealMsgWaitForMultipleObjectsEx ieui!CoreSC::Wait ieui!CoreSC::WaitMessage ieui!WaitMessageEx ieframe!CBrowserFrame::FrameMessagePump ieframe!BrowserThreadProc ieframe!BrowserNewThreadProc ieframe!SHOpenFolderWindow kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

35 Id: 136c.135c Suspend: 0 Teb: 7ff7c000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

36 Id: 136c.c34 Suspend: 0 Teb: 7ff7b000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

37 Id: 136c.a08 Suspend: 0 Teb: 7ff7a000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

38 Id: 136c.1108 Suspend: 0 Teb: 7ff79000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

39 Id: 136c.c20 Suspend: 0 Teb: 7ff77000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

40 Id: 136c.e48 Suspend: 0 Teb: 7ff74000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

41 Id: 136c.298 Suspend: 0 Teb: 7ff73000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

42 Id: 136c.11c8 Suspend: 0 Teb: 7ff72000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

43 Id: 136c.1180 Suspend: 0 Teb: 7ff68000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

44 Id: 136c.1750 Suspend: 0 Teb: 7ff66000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

45 Id: 136c.16d8 Suspend: 0 Teb: 7ff65000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

46 Id: 136c.15b8 Suspend: 0 Teb: 7ff64000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

47 Id: 136c.b88 Suspend: 0 Teb: 7ff6f000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

48 Id: 136c.434 Suspend: 0 Teb: 7ff6e000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

49 Id: 136c.16e8 Suspend: 0 Teb: 7ff6d000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

50 Id: 136c.f04 Suspend: 0 Teb: 7ff6c000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

51 Id: 136c.128c Suspend: 0 Teb: 7ff67000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

52 Id: 136c.1074 Suspend: 0 Teb: 7ff63000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

53 Id: 136c.2dc Suspend: 0 Teb: 7ff62000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

54 Id: 136c.172c Suspend: 0 Teb: 7ff60000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

55 Id: 136c.1240 Suspend: 0 Teb: 7ff69000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

56 Id: 136c.1604 Suspend: 0 Teb: 7ff61000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

57 Id: 136c.6a4 Suspend: 0 Teb: 7ff5f000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

58 Id: 136c.1258 Suspend: 0 Teb: 7ff5e000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

59 Id: 136c.8c0 Suspend: 0 Teb: 7ff5b000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

60 Id: 136c.868 Suspend: 0 Teb: 7ff58000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

61 Id: 136c.a54 Suspend: 0 Teb: 7ff57000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

62 Id: 136c.77c Suspend: 0 Teb: 7ff56000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

63 Id: 136c.1290 Suspend: 0 Teb: 7ff59000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

64 Id: 136c.1480 Suspend: 0 Teb: 7ff55000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

65 Id: 136c.1270 Suspend: 0 Teb: 7ff54000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

66 Id: 136c.b8c Suspend: 0 Teb: 7ff53000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

67 Id: 136c.167c Suspend: 0 Teb: 7ff92000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

68 Id: 136c.176c Suspend: 0 Teb: 7ff8e000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

69 Id: 136c.80c Suspend: 0 Teb: 7ff8b000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

70 Id: 136c.1570 Suspend: 0 Teb: 7ff8a000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

71 Id: 136c.e74 Suspend: 0 Teb: 7ff78000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

72 Id: 136c.1490 Suspend: 0 Teb: 7ff76000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

73 Id: 136c.d28 Suspend: 0 Teb: 7ff6a000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

74 Id: 136c.8d8 Suspend: 0 Teb: 7ff5c000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

75 Id: 136c.1064 Suspend: 0 Teb: 7ff4a000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

76 Id: 136c.1478 Suspend: 0 Teb: 7ff47000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

77 Id: 136c.1470 Suspend: 0 Teb: 7ff44000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

78 Id: 136c.aa0 Suspend: 0 Teb: 7ff43000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

79 Id: 136c.1210 Suspend: 0 Teb: 7ff48000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

80 Id: 136c.954 Suspend: 0 Teb: 7ff46000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

81 Id: 136c.9d4 Suspend: 0 Teb: 7ff45000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

82 Id: 136c.f30 Suspend: 0 Teb: 7ff42000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

83 Id: 136c.cc4 Suspend: 0 Teb: 7ff34000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

84 Id: 136c.1018 Suspend: 0 Teb: 7ff33000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

85 Id: 136c.940 Suspend: 0 Teb: 7ff32000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

86 Id: 136c.bd8 Suspend: 0 Teb: 7ff31000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

87 Id: 136c.1714 Suspend: 0 Teb: 7ff84000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage user32!DialogBox2 user32!InternalDialogBox user32!SoftModalMessageBox user32!MessageBoxWorker user32!MessageBoxTimeoutW user32!MessageBoxTimeoutA user32!MessageBoxExA ieframe!Detour_MessageBoxExA user32!MessageBoxA WARNING: Stack unwind information not available. Following frames may be wrong. MathMLMimer!DllUnregisterServer MathMLMimer!DllUnregisterServer MathMLMimer!DllUnregisterServer

88 Id: 136c.1744 Suspend: 0 Teb: 7ff83000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

89 Id: 136c.9cc Suspend: 0 Teb: 7ff81000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

90 Id: 136c.f68 Suspend: 0 Teb: 7ff5a000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

91 Id: 136c.17f4 Suspend: 0 Teb: 7ff49000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

92 Id: 136c.13c Suspend: 0 Teb: 7ff41000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

93 Id: 136c.110 Suspend: 0 Teb: 7ff3f000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

94 Id: 136c.8e4 Suspend: 0 Teb: 7ff3e000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

95 Id: 136c.fcc Suspend: 0 Teb: 7ff4c000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx user32!RealMsgWaitForMultipleObjectsEx ieui!CoreSC::Wait ieui!CoreSC::WaitMessage ieui!WaitMessageEx ieframe!CBrowserFrame::FrameMessagePump ieframe!BrowserThreadProc ieframe!BrowserNewThreadProc ieframe!SHOpenFolderWindow kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

96 Id: 136c.1378 Suspend: 0 Teb: 7ff4b000 Unfrozen

ntdll!KiFastSystemCallRet user32!NtUserWaitMessage ieframe!CTabWindow::_TabWindowThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

97 Id: 136c.8ec Suspend: 0 Teb: 7ff40000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllGetClassObject ToolbarA!ToolBarMain ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

98 Id: 136c.ac Suspend: 0 Teb: 7ff3d000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

99 Id: 136c.79c Suspend: 0 Teb: 7ff3c000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects WARNING: Stack unwind information not available. Following frames may be wrong. ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ToolbarA!DllCanUnloadNow ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

100 Id: 136c.bd0 Suspend: 0 Teb: 7ff3a000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForMultipleObjects kernel32!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects msiltcfg!WorkerThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

101 Id: 136c.1504 Suspend: 0 Teb: 7ff36000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

102 Id: 136c.5c0 Suspend: 0 Teb: 7ff35000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

103 Id: 136c.17a0 Suspend: 0 Teb: 7ff2d000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

104 Id: 136c.17c4 Suspend: 0 Teb: 7ff2c000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

105 Id: 136c.f08 Suspend: 0 Teb: 7ffaa000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

106 Id: 136c.1268 Suspend: 0 Teb: 7ff28000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

107 Id: 136c.12c8 Suspend: 0 Teb: 7ff27000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

108 Id: 136c.1634 Suspend: 0 Teb: 7ff26000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

109 Id: 136c.120c Suspend: 0 Teb: 7ff21000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

110 Id: 136c.13f4 Suspend: 0 Teb: 7ff20000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

111 Id: 136c.1494 Suspend: 0 Teb: 7ff1f000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

112 Id: 136c.16ec Suspend: 0 Teb: 7ff1e000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus dxtrans!TMThreadProc kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

113 Id: 136c.d68 Suspend: 0 Teb: 7ffa4000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwRemoveIoCompletion kernel32!GetQueuedCompletionStatus rpcrt4!COMMON_ProcessCalls rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents rpcrt4!ProcessIOEventsWrapper rpcrt4!BaseCachedThreadRoutine rpcrt4!ThreadStartRoutine kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

114 Id: 136c.5fc Suspend: 0 Teb: 7ffdd000 Unfrozen

ntdll!KiFastSystemCallRet ntdll!ZwWaitForWorkViaWorkerFactory ntdll!TppWorkerThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

In contrast, Flash10b module (shown in magenta above) is not ubiquitous, it is present on just 2 threads.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | 2 Comments »

Succession of Patterns (Part 1)

Monday, June 22nd, 2009

Looking at pattern cooperation studies it is easy to see that some patterns precede others, for example, heap corruption might be blocked by a hard error or a message box and therefore block other threads, creating conditions for another pattern to appear, wait chains. Blocked threads may block other coupled processes creating inter-process wait chains. Successive patterns reach the climax at the end and the system is no longer able to generate any other patterns.

The goal here is to find patterns that most likely happen in succession and another pattern series that are less likely to effect other abnormal conditions. Such pattern sequences can help in troubleshooting and finding root causes.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Memory Analysis Forensics and Intelligence | 3 Comments »

Null data pointer, pass through functions and platformorphic fault: pattern cooperation

Saturday, June 20th, 2009

We got a bugcheck when accessing a NULL data pointer:

1: kd> r Last set context: rax=0000000063537852 rbx=0000000000000000 rcx=0000000000000009 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000 rip=fffffadf262760da rsp=fffffadf15973968 rbp=0000000070537852 r8=fffffadf31614b00 r9=fffffadffe9fa7b0 r10=000000000000000a r11=fffffadf31614bf0 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz na po nc cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 efl=00010206 rdbss!RxIsThisACscAgentOpen+0×30: fffffadf`262760da f3a6 repe cmps byte ptr [rsi],byte ptr [rdi]

Default analysis via !analyze -v pointed to the first non-MS module DriverA (the identification process is explained here) located on the following stack trace (that also shows exception processing in file system kernel drivers):

1: kd> kc 100 Call Site nt!KeBugCheckEx rdbss!RxExceptionFilter+0x15e rdbss!RxFsdCommonDispatch+0x6d5 nt!_C_specific_handler+0x9b nt!RtlpExecuteHandlerForException+0xd nt!RtlDispatchException+0x2c0 nt!KiDispatchException+0xd9 nt!KiExceptionExit nt!KiPageFault+0x1e1 rdbss!RxIsThisACscAgentOpen+0x30 rdbss!RxInitializeVNetRootParameters+0x31d rdbss!RxFindOrConstructVirtualNetRoot+0x180 rdbss!RxCanonicalizeNameAndObtainNetRoot+0x223 rdbss!RxCommonCreate+0x470 rdbss!RxFsdCommonDispatch+0x51c mrxsmb!MRxSmbFsdDispatch+0x211 fltmgr!FltpCreate+0x353 DriverA!DispatchThrough+0×177 DriverB!PassThrough+0×12c fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×3f2 fltmgr!FltpCreate+0×3bd Mup!DnrRedirectFileOpen+0×791 Mup!DnrNameResolve+0xb19 Mup!DnrStartNameResolution+0×478 Mup!DfsCommonCreate+0×3dc Mup!DfsFsdCreate+0×10d Mup!MupCreate+0×125 nt!IopParseDevice+0×1088 nt!ObpLookupObjectName+0×931 nt!ObOpenObjectByName+0×180 nt!IopCreateFile+0×630 nt!IoCreateFileSpecifyDeviceObjectHint+0xb4 fltmgr!FltpExpandFilePathWorker+0×23d fltmgr!FltpExpandFilePath+0×5b fltmgr!FltpGetOpenedDestinationFileName+0xd5 fltmgr!FltpGetNormalizedDestinationFileName+0×15 fltmgr!FltGetDestinationFileNameInformation+0×17d DriverC+0×383e fltmgr!FltpPerformPreCallbacks+0×3e2 fltmgr!FltpPassThroughInternal+0×40 fltmgr!FltpDispatch+0×102 Mup!DfsCommonSetInformation+0×165 Mup!DfsFsdSetInformation+0×67 nt!NtSetInformationFile+0×916 nt!KiSystemServiceCopyEnd+0×3

Although DriverA function on the stack trace looked like a pass through, DriverA.sys file was removed from the system. Nevertheless, the same pattern continued:

0: kd> kc 100 Call Site nt!KeBugCheckEx rdbss!RxExceptionFilter+0x15e rdbss!RxFsdCommonDispatch+0x6d5 nt!_C_specific_handler+0x9b nt!RtlpExecuteHandlerForException+0xd nt!RtlDispatchException+0x2c0 nt!KiDispatchException+0xd9 nt!KiExceptionExit nt!KiPageFault+0x1e1 rdbss!RxIsThisACscAgentOpen+0x30 rdbss!RxInitializeVNetRootParameters+0x31d rdbss!RxFindOrConstructVirtualNetRoot+0x180 rdbss!RxCanonicalizeNameAndObtainNetRoot+0x223 rdbss!RxCommonCreate+0x470 rdbss!RxFsdCommonDispatch+0x51c mrxsmb!MRxSmbFsdDispatch+0x211 fltmgr!FltpCreate+0x353 DriverB!PassThrough+0×12c fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×3f2 fltmgr!FltpCreate+0×3bd Mup!DnrRedirectFileOpen+0×791 Mup!DnrNameResolve+0xb19 Mup!DnrStartNameResolution+0×478 Mup!DfsCommonCreate+0×3dc Mup!DfsFsdCreate+0×10d Mup!MupCreate+0×125 nt!IopParseDevice+0×1088 nt!ObpLookupObjectName+0×931 nt!ObOpenObjectByName+0×180 nt!IopCreateFile+0×630 nt!IoCreateFileSpecifyDeviceObjectHint+0xb4 fltmgr!FltpExpandFilePathWorker+0×23d fltmgr!FltpExpandFilePath+0×5b fltmgr!FltpGetOpenedDestinationFileName+0xd5 fltmgr!FltpGetNormalizedDestinationFileName+0×15 fltmgr!FltGetDestinationFileNameInformation+0×17d DriverC+0×383e fltmgr!FltpPerformPreCallbacks+0×3e2 fltmgr!FltpPassThroughInternal+0×40 fltmgr!FltpDispatch+0×102 Mup!DfsCommonSetInformation+0×165 Mup!DfsFsdSetInformation+0×67 nt!NtSetInformationFile+0×916 nt!KiSystemServiceCopyEnd+0×3

So it was concluded that the presence of DriverA was irrelevant to the problem. Now DriverB was pointed to by the default analysis as a possible culprit. However, the fault appeared platformorphic: Google search found another similar stack trace shape with the same faulted instruction but without DriverB and DriverC. Therefore the conclusion was that modules DriverA, DriverB and DriverC didn’t have the straightforward contribution to the abnormal system behaviour.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

June 2026
M	T	W	T	F	S	S
« May
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

Archive for the ‘Crash Dump Patterns’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta