Crash Dump Analysis Patterns (Part 11)

One of mistakes beginners make is trusting WinDbg !analyze or kv commands displaying stack trace. WinDbg is only a tool, sometimes information necessary to get correct stack trace is missing and therefore some critical thought is required to distinguish between correct and incorrect stack traces. I call this pattern Incorrect Stack Trace. Incorrect stack traces usually

  • Have WinDbg warning: “Following frames may be wrong”

  • Don’t have the correct bottom frame like kernel32!BaseThreadStart (in user-mode)

  • Have function calls that don’t make any sense

  • Have strange looking disassembled function code or code that doesn’t make any sense from compiler perspective

  • Have ChildEBP and RetAddr addresses that don’t make any sense

Consider the following stack trace:

0:011> k
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0184e434 7c830b10 0×184e5bf
0184e51c 7c81f832 ntdll!RtlGetFullPathName_Ustr+0×15b
0184e5f8 7c83b1dd ntdll!RtlpLowFragHeapAlloc+0xc6a
00099d30 00000000 ntdll!RtlpLowFragHeapFree+0xa7

Here we have almost all attributes of the wrong stack trace. At the first glance it looks like some heap corruption happened (runtime heap alloc and free functions are present) but if you give it second thought you would see that low fragmentation heap Free function shouldn’t call low fragmentation heap Alloc function and the latter shoudn’t query full path name. That doesn’t make any sense.  

What we should do here? Look at raw stack and try to build the correct stack trace ourselves. In our case this is very easy. We need to traverse stack frames from BaseThreadStart+0×34 until we don’t find any function call or reach the top. When functions are called (no optimization, most compilers) EBP registers are linked together as explained on slide 13 here:

Practical Foundations of Debugging (6.1)

0:011> !teb
TEB at 7ffd8000
    ExceptionList:        0184ebdc
    StackBase:            01850000
    StackLimit:           01841000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffd8000
    EnvironmentPointer:   00000000
    ClientId:             0000061c . 00001b60
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffdf000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

0:011> dds 01841000 01850000
01841000  00000000



0184eef0  0184ef0c
0184eef4  7615dff2 localspl!SplDriverEvent+0×21
0184eef8  00bc3e08
0184eefc  00000003
0184ef00  00000001
0184ef04  00000000
0184ef08  0184efb0
0184ef0c  0184ef30
0184ef10  7615f9d0 localspl!PrinterDriverEvent+0×46
0184ef14  00bc3e08
0184ef18  00000003
0184ef1c  00000000
0184ef20  0184efb0
0184ef24  00b852a8
0184ef28  00c3ec58
0184ef2c  00bafcc0
0184ef30  0184f3f8
0184ef34  7614a9b4 localspl!SplAddPrinter+0×5f3
0184ef38  00c3ec58
0184ef3c  00000003
0184ef40  00000000
0184ef44  0184efb0
0184ef48  00c117f8



0184ff28  00000000
0184ff2c  00000000
0184ff30  0184ff84
0184ff34  77c75286 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×3a
0184ff38  0184ff4c
0184ff3c  77c75296 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×4a
0184ff40  7c82f2fc ntdll!RtlLeaveCriticalSection
0184ff44  000de378
0184ff48  00097df0
0184ff4c  4d2fa200
0184ff50  ffffffff
0184ff54  ca5b1700
0184ff58  ffffffff
0184ff5c  8082d821
0184ff60  0184fe38
0184ff64  00097df0
0184ff68  000000aa
0184ff6c  80020000
0184ff70  0184ff54
0184ff74  80020000
0184ff78  000b0c78
0184ff7c  00a50180
0184ff80  0184fe38
0184ff84  0184ff8c
0184ff88  77c5778f RPCRT4!RecvLotsaCallsWrapper+0xd
0184ff8c  0184ffac
0184ff90  77c5f7dd RPCRT4!BaseCachedThreadRoutine+0×9d
0184ff94  0009c410
0184ff98  00000000
0184ff9c  00000000
0184ffa0  00097df0
0184ffa4  00097df0
0184ffa8  00015f90
0184ffac  0184ffb8
0184ffb0  77c5de88 RPCRT4!ThreadStartRoutine+0×1b
0184ffb4  00088258
0184ffb8  0184ffec
0184ffbc  77e6608b kernel32!BaseThreadStart+0×34
0184ffc0  00097df0
0184ffc4  00000000
0184ffc8  00000000
0184ffcc  00097df0
0184ffd0  8ad84818
0184ffd4  0184ffc4
0184ffd8  8980a700
0184ffdc  ffffffff
0184ffe0  77e6b7d0 kernel32!_except_handler3
0184ffe4  77e66098 kernel32!`string’+0×98
0184ffe8  00000000
0184ffec  00000000
0184fff0  00000000
77c5de6d  RPCRT4!ThreadStartRoutine
0184fff8  00097df0
0184fffc  00000000
01850000  00000008

Next we need to use custom k command and specify base pointer. In our case the last found stack address that links EBP pointers is 0184eef0:

0:011> k L=0184eef0
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0184eef0 7615dff2 0×184e5bf
0184ef0c 7615f9d0 localspl!SplDriverEvent+0×21
0184ef30 7614a9b4 localspl!PrinterDriverEvent+0×46
0184f3f8 761482de localspl!SplAddPrinter+0×5f3
0184f424 74067c8f localspl!LocalAddPrinterEx+0×2e
0184f874 74067b76 SPOOLSS!AddPrinterExW+0×151
0184f890 01007e29 SPOOLSS!AddPrinterW+0×17
0184f8ac 01006ec3 spoolsv!YAddPrinter+0×75
0184f8d0 77c70f3b spoolsv!RpcAddPrinter+0×37
0184f8f8 77ce23f7 RPCRT4!Invoke+0×30
0184fcf8 77ce26ed RPCRT4!NdrStubCall2+0×299
0184fd14 77c709be RPCRT4!NdrServerCall2+0×19
0184fd48 77c7093f RPCRT4!DispatchToStubInCNoAvrf+0×38
0184fd9c 77c70865 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×117
0184fdc0 77c734b1 RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3
0184fdfc 77c71bb3 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×42c
0184fe20 77c75458 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0184ff84 77c5778f RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0184ff8c 77c5f7dd RPCRT4!RecvLotsaCallsWrapper+0xd

Stack traces make more sense now but we don’t see BaseThreadStart+0×34. By default WinDbg displays only certain amount of function calls (stack frames) so we need to specify stack frame count, for example, 100:

0:011> k L=0184eef0 100
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0184eef0 7615dff2 0×184e5bf
0184ef0c 7615f9d0 localspl!SplDriverEvent+0×21
0184ef30 7614a9b4 localspl!PrinterDriverEvent+0×46
0184f3f8 761482de localspl!SplAddPrinter+0×5f3
0184f424 74067c8f localspl!LocalAddPrinterEx+0×2e
0184f874 74067b76 SPOOLSS!AddPrinterExW+0×151
0184f890 01007e29 SPOOLSS!AddPrinterW+0×17
0184f8ac 01006ec3 spoolsv!YAddPrinter+0×75
0184f8d0 77c70f3b spoolsv!RpcAddPrinter+0×37
0184f8f8 77ce23f7 RPCRT4!Invoke+0×30
0184fcf8 77ce26ed RPCRT4!NdrStubCall2+0×299
0184fd14 77c709be RPCRT4!NdrServerCall2+0×19
0184fd48 77c7093f RPCRT4!DispatchToStubInCNoAvrf+0×38
0184fd9c 77c70865 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×117
0184fdc0 77c734b1 RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3
0184fdfc 77c71bb3 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×42c
0184fe20 77c75458 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0184ff84 77c5778f RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0184ff8c 77c5f7dd RPCRT4!RecvLotsaCallsWrapper+0xd
0184ffac 77c5de88 RPCRT4!BaseCachedThreadRoutine+0×9d
0184ffb8 77e6608b RPCRT4!ThreadStartRoutine+0×1b
0184ffec 00000000 kernel32!BaseThreadStart+0×34

Now stack trace looks much better. 

- Dmitry Vostokov @ DumpAnalysis.org -

22 Responses to “Crash Dump Analysis Patterns (Part 11)”

  1. Dmitry Vostokov Says:

    I’ve published another case study for this pattern with complete source code:

    http://www.dumpanalysis.org/blog/index.php/2007/07/25/reconstructing-stack-trace-manually/

  2. abhi Says:

    Hello,
    My dump analysis looks like this. This is a hang dump file of a COM+ component on Windows server 2003 SP2.
    Can you something obvious here. appreciate your help. Thanks

    # 0 Id: f34.f40 Suspend: 1 Teb: 7ffdf000 Unfrozen
    ChildEBP RetAddr Args to Child
    0007fcd4 7c827d0b 77e61d1e 00000078 00000000 ntdll!KiFastSystemCallRet
    0007fcd8 77e61d1e 00000078 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0007fd48 77e61c8d 00000078 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0x88
    0007fd5c 77727b21 00000078 ffffffff ffffffff kernel32!WaitForSingleObject+0x12
    0007fd78 77728683 0009a6b0 0007fdc7 00000000 ole32!CSurrogateProcessActivator::WaitForSurrogateTimeout+0x55
    0007fd90 01001383 0007fdac 00000000 00000000 ole32!CoRegisterSurrogateEx+0x1dd
    0007ff1c 010016a1 01000000 00000000 000924b0 dllhost!WinMain+0xe3
    0007ffc0 77e6f23b 00000000 00000000 7ffdc000 dllhost!WinMainCRTStartup+0x182
    0007fff0 00000000 0100151f 00000000 78746341 kernel32!ProcessIdToSessionId+0x209

    1 Id: f34.f2c Suspend: 1 Teb: 7ffdd000 Unfrozen
    ChildEBP RetAddr Args to Child
    00c2fe18 7c82783b 77c885ac 00000120 00c2ff74 ntdll!KiFastSystemCallRet
    00c2fe1c 77c885ac 00000120 00c2ff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc
    00c2ff84 77c88792 00c2ffac 77c8872d 0009fa58 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
    00c2ff8c 77c8872d 0009fa58 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
    00c2ffac 77c7b110 0009cdc0 00c2ffec 77e64829 rpcrt4!BaseCachedThreadRoutine+0x9d
    00c2ffb8 77e64829 0009fc80 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00c2ffec 00000000 77c7b0f5 0009fc80 00000000 kernel32!GetModuleHandleA+0xdf

    2 Id: f34.f94 Suspend: 1 Teb: 7ffda000 Unfrozen
    ChildEBP RetAddr Args to Child
    00caff0c 7739c811 7739c844 00caff50 00000000 ntdll!KiFastSystemCallRet
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00caff2c 7768ffb3 00caff50 00000000 00000000 user32!PostMessageW+0x39
    00caff6c 7768f366 00007530 77e61c96 000a26d8 ole32!CDllHost::STAWorkerLoop+0x72
    00caff88 7768f2a2 00caffac 776bbab4 77792c30 ole32!CDllHost::WorkerThread+0xc8
    00caff90 776bbab4 77792c30 00000000 000a26d8 ole32!DLLHostThreadEntry+0xd
    00caffac 776b1704 00000000 00caffec 77e64829 ole32!CRpcThread::WorkerLoop+0x26
    00caffb8 77e64829 000a26d8 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
    00caffec 00000000 776b16e4 000a26d8 00000000 kernel32!GetModuleHandleA+0xdf

    3 Id: f34.f98 Suspend: 1 Teb: 7ffd8000 Unfrozen
    ChildEBP RetAddr Args to Child
    00d7ff9c 7c826f4b 7c83d424 00000001 00d7ffb0 ntdll!KiFastSystemCallRet
    00d7ffa0 7c83d424 00000001 00d7ffb0 00000000 ntdll!NtDelayExecution+0xc
    00d7ffb8 77e64829 00000000 00000000 00000000 ntdll!RtlpTimerThread+0x47
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00d7ffec 00000000 7c83d3dd 00000000 00000000 kernel32!GetModuleHandleA+0xdf

    4 Id: f34.f90 Suspend: 1 Teb: 7ffd7000 Unfrozen
    ChildEBP RetAddr Args to Child
    00dbff70 7c8277db 7c839f38 0000017c 00dbffb0 ntdll!KiFastSystemCallRet
    00dbff74 7c839f38 0000017c 00dbffb0 00dbffb4 ntdll!ZwRemoveIoCompletion+0xc
    00dbffb8 77e64829 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x3d
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00dbffec 00000000 7c839efb 00000000 00000000 kernel32!GetModuleHandleA+0xdf

    5 Id: f34.f70 Suspend: 1 Teb: 7ffd4000 Unfrozen
    ChildEBP RetAddr Args to Child
    00fefee8 7c826f4b 77e41ed1 00000000 00feff2c ntdll!KiFastSystemCallRet
    00fefeec 77e41ed1 00000000 00feff2c 00000000 ntdll!NtDelayExecution+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00feff54 77e424ed 000003e8 00000000 00feffb8 kernel32!SleepEx+0x4d
    00feff64 4a77de1e 000003e8 00000000 00000000 kernel32!Sleep+0xf
    00feffb8 77e64829 00000000 00000000 00000000 comsvcs!PostData+0x1dd
    00feffec 00000000 4a77dc41 00000000 00000000 kernel32!GetModuleHandleA+0xdf

    6 Id: f34.f64 Suspend: 1 Teb: 7ffaf000 Unfrozen
    ChildEBP RetAddr Args to Child
    0104fee8 7c827d0b 77e61d1e 000001ec 00000000 ntdll!KiFastSystemCallRet
    0104feec 77e61d1e 000001ec 00000000 0104ff30 ntdll!NtWaitForSingleObject+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0104ff5c 77e61c8d 000001ec 00002710 00000000 kernel32!WaitForSingleObjectEx+0x88
    0104ff70 4a77921b 000001ec 00002710 00037988 kernel32!WaitForSingleObject+0x12
    0104ff84 77bcb530 4a8098d0 00000000 00000000 comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop+0x29
    0104ffb8 77e64829 00037650 00000000 00000000 msvcrt!_endthreadex+0xa3
    0104ffec 00000000 77bcb4bc 00037650 00000000 kernel32!GetModuleHandleA+0xdf

    7 Id: f34.f7c Suspend: 1 Teb: 7ffae000 Unfrozen
    ChildEBP RetAddr Args to Child
    0108fee0 7c827d0b 77e61d1e 000001ec 00000000 ntdll!KiFastSystemCallRet
    0108fee4 77e61d1e 000001ec 00000000 0108ff28 ntdll!NtWaitForSingleObject+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0108ff54 77e61c8d 000001ec 00003a98 00000000 kernel32!WaitForSingleObjectEx+0x88
    0108ff68 4a778cb6 000001ec 00003a98 00000000 kernel32!WaitForSingleObject+0x12
    0108ff84 77bcb530 4a8098d0 00000000 00000000 comsvcs!CSTAThreadPool::KillThreadControlLoop+0x25
    0108ffb8 77e64829 000376e8 00000000 00000000 msvcrt!_endthreadex+0xa3
    0108ffec 00000000 77bcb4bc 000376e8 00000000 kernel32!GetModuleHandleA+0xdf

    8 Id: f34.f80 Suspend: 1 Teb: 7ffad000 Unfrozen
    ChildEBP RetAddr Args to Child
    010cfdcc 7c827cfb 77e6202c 00000003 010cfe1c ntdll!KiFastSystemCallRet
    010cfdd0 77e6202c 00000003 010cfe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    010cfe78 7739bbd1 00000003 010cfea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    010cfed4 7739ce36 00000002 010cff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    010cfef0 4a77cb28 00000002 010cff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    010cff84 77bcb530 000c14a0 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    010cffb8 77e64829 00037780 00000000 00000000 msvcrt!_endthreadex+0xa3
    010cffec 00000000 77bcb4bc 00037780 00000000 kernel32!GetModuleHandleA+0xdf

    9 Id: f34.f68 Suspend: 1 Teb: 7ffac000 Unfrozen
    ChildEBP RetAddr Args to Child
    0110fec8 7c8277db 77e5bea2 00000188 0110ff50 ntdll!KiFastSystemCallRet
    0110fecc 77e5bea2 00000188 0110ff50 0110ff10 ntdll!ZwRemoveIoCompletion+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0110fef8 4a7ef40f 00000188 0110ff38 0110ff50 kernel32!GetQueuedCompletionStatus+0x29
    0110ff74 4a7ef6aa 000addc0 00037b50 0110ffb8 comsvcs!WORK_QUEUE::WorkerLoop+0x66
    0110ff84 77bcb530 000addc0 00000000 00000000 comsvcs!WORK_QUEUE::ThreadLoop+0x19
    0110ffb8 77e64829 00036e40 00000000 00000000 msvcrt!_endthreadex+0xa3
    0110ffec 00000000 77bcb4bc 00036e40 00000000 kernel32!GetModuleHandleA+0xdf

    10 Id: f34.f54 Suspend: 1 Teb: 7ffab000 Unfrozen
    ChildEBP RetAddr Args to Child
    0114fdcc 7c827cfb 77e6202c 00000003 0114fe1c ntdll!KiFastSystemCallRet
    0114fdd0 77e6202c 00000003 0114fe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0114fe78 7739bbd1 00000003 0114fea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    0114fed4 7739ce36 00000002 0114ff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    0114fef0 4a77cb28 00000002 0114ff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    0114ff84 77bcb530 000c45f0 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    0114ffb8 77e64829 00037780 00000000 00000000 msvcrt!_endthreadex+0xa3
    0114ffec 00000000 77bcb4bc 00037780 00000000 kernel32!GetModuleHandleA+0xdf

    11 Id: f34.f5c Suspend: 1 Teb: 7ffaa000 Unfrozen
    ChildEBP RetAddr Args to Child
    0118fdcc 7c827cfb 77e6202c 00000003 0118fe1c ntdll!KiFastSystemCallRet
    0118fdd0 77e6202c 00000003 0118fe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0118fe78 7739bbd1 00000003 0118fea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    0118fed4 7739ce36 00000002 0118ff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    0118fef0 4a77cb28 00000002 0118ff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    0118ff84 77bcb530 000c81c8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    0118ffb8 77e64829 00037780 00000000 00000000 msvcrt!_endthreadex+0xa3
    0118ffec 00000000 77bcb4bc 00037780 00000000 kernel32!GetModuleHandleA+0xdf

    12 Id: f34.f48 Suspend: 1 Teb: 7ffa9000 Unfrozen
    ChildEBP RetAddr Args to Child
    011cfdcc 7c827cfb 77e6202c 00000003 011cfe1c ntdll!KiFastSystemCallRet
    011cfdd0 77e6202c 00000003 011cfe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    011cfe78 7739bbd1 00000003 011cfea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    011cfed4 7739ce36 00000002 011cff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    011cfef0 4a77cb28 00000002 011cff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    011cff84 77bcb530 000c8aa8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    011cffb8 77e64829 00037780 00000000 00000000 msvcrt!_endthreadex+0xa3
    011cffec 00000000 77bcb4bc 00037780 00000000 kernel32!GetModuleHandleA+0xdf

    13 Id: f34.f78 Suspend: 1 Teb: 7ffd5000 Unfrozen
    ChildEBP RetAddr Args to Child
    00fafdcc 7c827cfb 77e6202c 00000003 00fafe1c ntdll!KiFastSystemCallRet
    00fafdd0 77e6202c 00000003 00fafe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00fafe78 7739bbd1 00000003 00fafea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    00fafed4 7739ce36 00000002 00faff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    00fafef0 4a77cb28 00000002 00faff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    00faff84 77bcb530 000c7760 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    00faffb8 77e64829 00037818 00000000 00000000 msvcrt!_endthreadex+0xa3
    00faffec 00000000 77bcb4bc 00037818 00000000 kernel32!GetModuleHandleA+0xdf

    14 Id: f34.f88 Suspend: 1 Teb: 7ffa8000 Unfrozen
    ChildEBP RetAddr Args to Child
    0120fdcc 7c827cfb 77e6202c 00000003 0120fe1c ntdll!KiFastSystemCallRet
    0120fdd0 77e6202c 00000003 0120fe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0120fe78 7739bbd1 00000003 0120fea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    0120fed4 7739ce36 00000002 0120ff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    0120fef0 4a77cb28 00000002 0120ff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    0120ff84 77bcb530 000c92a8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    0120ffb8 77e64829 00037818 00000000 00000000 msvcrt!_endthreadex+0xa3
    0120ffec 00000000 77bcb4bc 00037818 00000000 kernel32!GetModuleHandleA+0xdf

    15 Id: f34.f8c Suspend: 1 Teb: 7ffa7000 Unfrozen
    ChildEBP RetAddr Args to Child
    0124fdcc 7c827cfb 77e6202c 00000003 0124fe1c ntdll!KiFastSystemCallRet
    0124fdd0 77e6202c 00000003 0124fe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0124fe78 7739bbd1 00000003 0124fea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    0124fed4 7739ce36 00000002 0124ff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    0124fef0 4a77cb28 00000002 0124ff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    0124ff84 77bcb530 000c9928 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    0124ffb8 77e64829 00037818 00000000 00000000 msvcrt!_endthreadex+0xa3
    0124ffec 00000000 77bcb4bc 00037818 00000000 kernel32!GetModuleHandleA+0xdf

    16 Id: f34.f74 Suspend: 1 Teb: 7ffa6000 Unfrozen
    ChildEBP RetAddr Args to Child
    0128fdcc 7c827cfb 77e6202c 00000003 0128fe1c ntdll!KiFastSystemCallRet
    0128fdd0 77e6202c 00000003 0128fe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0128fe78 7739bbd1 00000003 0128fea0 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
    0128fed4 7739ce36 00000002 0128ff74 ffffffff user32!MsgWaitForMultipleObjectsEx+0xd7
    0128fef0 4a77cb28 00000002 0128ff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
    0128ff84 77bcb530 000ca8e0 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
    0128ffb8 77e64829 00037818 00000000 00000000 msvcrt!_endthreadex+0xa3
    0128ffec 00000000 77bcb4bc 00037818 00000000 kernel32!GetModuleHandleA+0xdf

    17 Id: f34.e24 Suspend: 1 Teb: 7ffa5000 Unfrozen
    ChildEBP RetAddr Args to Child
    0268fd00 7c827d0b 77e61d1e 00000394 00000000 ntdll!KiFastSystemCallRet
    0268fd04 77e61d1e 00000394 00000000 0268fd48 ntdll!NtWaitForSingleObject+0xc
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0268fd74 77e61c8d 00000394 00002710 00000000 kernel32!WaitForSingleObjectEx+0x88
    0268fd88 4a756c3b 00000394 00002710 00000000 kernel32!WaitForSingleObject+0x12
    0268ffb8 77e64829 000a7610 00000000 00000000 comsvcs!PingThread+0xf6
    0268ffec 00000000 4a756b45 000a7610 00000000 kernel32!GetModuleHandleA+0xdf

  3. Dmitry Vostokov Says:

    Are you sure you have correct symbols applied? You have

    WARNING: Stack unwind information not available.

    always everywhere. Maybe the latest WinDbg will pick up them correctly. I had the similar problem with W2K3 R02 when I was using the previous version of WinDbg. I’m not an expert in COM+ hang dumps, it is better to ask MS support if !analyze -v -hang doesn’t point to the right direction.
    There is no 3rd-party code visible in stack traces so it might be purely MS COM+ infrastructure problem or the problem in MS COM+ component.

  4. Dmitry Vostokov Says:

    Sometimes incorrect stack trace is reported when symbols were not applied. Non-symbol gaps in stack traces can be the sign of this pattern too.

    STACK_TEXT:
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00b2f42c 091607aa mydll!foo+0×8338
    00b2f4cc 7c83ab9e mydll!foo+0×8fe3
    00b2f4ec 7c832d06 ntdll!RtlFindNextActivationContextSection+0×46
    00b2f538 001a5574 ntdll!RtlFindActivationContextSectionString+0xe1
    00b2f554 7c8302b3 0×1a5574
    00b2f560 7c82f9c1 ntdll!RtlpFreeToHeapLookaside+0×22
    00b2f640 7c832b7f ntdll!RtlFreeHeap+0×20e
    001dd000 00080040 ntdll!LdrUnlockLoaderLock+0xad
    001dd00c 0052005c 0×80040
    001dd010 00470045 0×52005c
    0052005c 00000000 0×470045

  5. Steven Says:

    Hi, i’m using windbg, because i AM a beginner and don’t have a clue about analyzing minidumps…

    this is mine, i hope you maybe can get some information out of it, as for me i think it may be the video card…but it worked perfect on other computers. Memtest passed 11 times…i don’t know what the problem is :(

    0: kd> !analyze -v

    PAGE_FAULT_IN_NONPAGED_AREA (50)
    Invalid system memory was referenced. This cannot be protected by try-except,
    it must be protected by a Probe. Typically the address is just plain bad or it
    is pointing at freed memory.
    Arguments:
    Arg1: e3421304, memory referenced.
    Arg2: 00000000, value 0 = read operation, 1 = write operation.
    Arg3: bf8afed3, If non-zero, the instruction address which referenced the bad memory
    address.
    Arg4: 00000001, (reserved)

    Debugging Details:
    ——————

    Could not read faulting driver name

    READ_ADDRESS: e3421304

    FAULTING_IP:
    win32k!WatchdogDrvGetDirectDrawInfo+b1
    bf8afed3 8b0e mov ecx,dword ptr [esi]

    MM_INTERNAL_CODE: 1

    CUSTOMER_CRASH_COUNT: 1

    DEFAULT_BUCKET_ID: DRIVER_FAULT

    BUGCHECK_STR: 0×50

    PROCESS_NAME: hl.exe

    LAST_CONTROL_TRANSFER: from bf008af2 to bf8afed3

    STACK_TEXT:
    aac329c0 bf008af2 e5347080 e5657754 aac329fc win32k!WatchdogDrvGetDirectDrawInfo+0xb1
    aac32a08 bf009033 00000001 e19bf680 aac32a90 dxg!vDdEnableDriver+0×8a
    aac32a24 bf892b95 e5657008 00000001 e3fda008 dxg!DxDdEnableDirectDraw+0xbf
    aac32a44 bf8ad4e5 00000000 e1856ad0 00000000 win32k!PDEVOBJ::bMakeSurface+0xe2
    aac32a70 bf8adee5 e1856ad0 e59ec828 e4882d18 win32k!hCreateHDEV+0×3a9
    aac32be8 bf8b4828 aac32cb0 e5a68400 00000001 win32k!DrvCreateMDEV+0×4dc
    aac32cdc bf8b6a8d aac32cb0 e160c2d8 00000000 win32k!DrvChangeDisplaySettings+0×251
    aac32d20 bf8b6980 00000000 0017ae60 00000000 win32k!xxxUserChangeDisplaySettings+0×141
    aac32d48 805409ac 00000000 0017ae60 00000000 win32k!NtUserChangeDisplaySettings+0×4a
    aac32d48 7c91eb94 00000000 0017ae60 00000000 nt!KiFastCallEntry+0xfc
    WARNING: Frame IP not in any known module. Following frames may be wrong.
    0012facc 00000000 00000000 00000000 00000000 0×7c91eb94

    STACK_COMMAND: kb

    FOLLOWUP_IP:
    win32k!WatchdogDrvGetDirectDrawInfo+b1
    bf8afed3 8b0e mov ecx,dword ptr [esi]

    SYMBOL_STACK_INDEX: 0

    SYMBOL_NAME: win32k!WatchdogDrvGetDirectDrawInfo+b1

    FOLLOWUP_NAME: MachineOwner

    MODULE_NAME: win32k

    IMAGE_NAME: win32k.sys

    DEBUG_FLR_IMAGE_TIMESTAMP: 45f0146f

    FAILURE_BUCKET_ID: 0×50_win32k!WatchdogDrvGetDirectDrawInfo+b1

    BUCKET_ID: 0×50_win32k!WatchdogDrvGetDirectDrawInfo+b1

    Followup: MachineOwner
    ———

  6. Dmitry Vostokov Says:

    May be a problem related to DirectDraw dxg.sys and possibly Watchdog drivers. Is stack trace consistent among various minidumps from this computer? Check the date of dxg.sys and any updates from MS.

  7. michaela Says:

    Hi,

    Can someone help me to explain whats happening on my server. It keep on crashing.

    Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible
    Product: Server, suite: Enterprise TerminalServer SingleUserTS
    Built by: 3790.srv03_sp2_rtm.070216-1710
    Kernel base = 0×80800000 PsLoadedModuleList = 0×808a6ea8
    Debug session time: Wed May 21 23:55:16.743 2008 (GMT+8)
    System Uptime: 1 days 5:48:24.125
    Loading Kernel Symbols
    Loading User Symbols
    Loading unloaded module list
    *
    * Bugcheck Analysis
    *

    Use !analyze -v to get detailed debugging information.

    BugCheck C2, {7, 121a, 0, 8b6e6d00}

    Probably caused by : Fs_Rec.SYS ( Fs_Rec!UdfsRecFsControl+63 )

    Followup: MachineOwner
    ———

    3: kd> !analyze -v;r;kv;lmtn;.logclose;q
    *
    * Bugcheck Analysis
    *

    BAD_POOL_CALLER (c2)
    The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
    Arguments:
    Arg1: 00000007, Attempt to free pool which was already freed
    Arg2: 0000121a, (reserved)
    Arg3: 00000000, Memory contents of the pool block
    Arg4: 8b6e6d00, Address of the block of pool being deallocated

    Debugging Details:
    ——————

    POOL_ADDRESS: 8b6e6d00

    FREED_POOL_TAG: Thre

    BUGCHECK_STR: 0xc2_7_Thre

    CUSTOMER_CRASH_COUNT: 1

    DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

    PROCESS_NAME: Rtvscan.exe

    CURRENT_IRQL: 0

    LAST_CONTROL_TRANSFER: from 808927bb to 80827c63

    STACK_TEXT:
    b86e18c0 808927bb 000000c2 00000007 0000121a nt!KeBugCheckEx+0×1b
    b86e1928 8081e1b6 8b6e6d00 00000000 8b6e6af8 nt!ExFreePoolWithTag+0×477
    b86e1954 f78037a1 8c8c8af8 8b95d030 b86e1988 nt!IopfCompleteRequest+0×180
    b86e1964 f780309e 8c8c8af8 8a3aabd8 8c897730 Fs_Rec!UdfsRecFsControl+0×63
    b86e1974 8081df65 8c8c8af8 8a3aabd8 8a3aabd8 Fs_Rec!FsRecFsControl+0×5a
    b86e1988 808f785c 80a5a4d0 8b95d030 80a5a540 nt!IofCallDriver+0×45
    b86e19d8 808220a4 8c8c8af8 b86e1c00 00000000 nt!IopMountVolume+0×1b4
    b86e1a04 808f8910 b86e1c38 8b95d000 b86e1b40 nt!IopCheckVpbMounted+0×5c
    b86e1afc 80937942 8b95d030 00000000 8a8d01e0 nt!IopParseDevice+0×3d4
    b86e1b7c 80933a76 00000000 b86e1bbc 00000040 nt!ObpLookupObjectName+0×5b0
    b86e1bd0 808ec76b 00000000 00000000 b86e1c01 nt!ObOpenObjectByName+0xea
    b86e1d54 8088978c 05e7e2b4 05e7e28c 05e7e2d4 nt!NtQueryAttributesFile+0×11d
    b86e1d54 7c8285ec 05e7e2b4 05e7e28c 05e7e2d4 nt!KiFastCallEntry+0xfc
    WARNING: Frame IP not in any known module. Following frames may be wrong.
    05e7e2d4 00000000 00000000 00000000 00000000 0×7c8285ec

    STACK_COMMAND: kb

    FOLLOWUP_IP:
    Fs_Rec!UdfsRecFsControl+63
    f78037a1 8bc6 mov eax,esi

    SYMBOL_STACK_INDEX: 3

    SYMBOL_NAME: Fs_Rec!UdfsRecFsControl+63

    FOLLOWUP_NAME: MachineOwner

    MODULE_NAME: Fs_Rec

    IMAGE_NAME: Fs_Rec.SYS

    DEBUG_FLR_IMAGE_TIMESTAMP: 3e800074

    FAILURE_BUCKET_ID: 0xc2_7_Thre_Fs_Rec!UdfsRecFsControl+63

    BUCKET_ID: 0xc2_7_Thre_Fs_Rec!UdfsRecFsControl+63

    Followup: MachineOwner
    ———

    eax=f773713c ebx=0000121a ecx=00000000 edx=8b6e6d00 esi=f7737120 edi=000001ff
    eip=80827c63 esp=b86e18a8 ebp=b86e18c0 iopl=0 nv up ei ng nz na pe nc
    cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000286
    nt!KeBugCheckEx+0×1b:
    80827c63 5d pop ebp
    ChildEBP RetAddr Args to Child
    b86e18c0 808927bb 000000c2 00000007 0000121a nt!KeBugCheckEx+0×1b (FPO: [Non-Fpo])
    b86e1928 8081e1b6 8b6e6d00 00000000 8b6e6af8 nt!ExFreePoolWithTag+0×477 (FPO: [Non-Fpo])
    b86e1954 f78037a1 8c8c8af8 8b95d030 b86e1988 nt!IopfCompleteRequest+0×180 (FPO: [Non-Fpo])
    b86e1964 f780309e 8c8c8af8 8a3aabd8 8c897730 Fs_Rec!UdfsRecFsControl+0×63 (FPO: [Non-Fpo])
    b86e1974 8081df65 8c8c8af8 8a3aabd8 8a3aabd8 Fs_Rec!FsRecFsControl+0×5a (FPO: [2,0,0])
    b86e1988 808f785c 80a5a4d0 8b95d030 80a5a540 nt!IofCallDriver+0×45 (FPO: [Non-Fpo])
    b86e19d8 808220a4 8c8c8af8 b86e1c00 00000000 nt!IopMountVolume+0×1b4 (FPO: [Non-Fpo])
    b86e1a04 808f8910 b86e1c38 8b95d000 b86e1b40 nt!IopCheckVpbMounted+0×5c (FPO: [Non-Fpo])
    b86e1afc 80937942 8b95d030 00000000 8a8d01e0 nt!IopParseDevice+0×3d4 (FPO: [Non-Fpo])
    b86e1b7c 80933a76 00000000 b86e1bbc 00000040 nt!ObpLookupObjectName+0×5b0 (FPO: [Non-Fpo])
    b86e1bd0 808ec76b 00000000 00000000 b86e1c01 nt!ObOpenObjectByName+0xea (FPO: [Non-Fpo])
    b86e1d54 8088978c 05e7e2b4 05e7e28c 05e7e2d4 nt!NtQueryAttributesFile+0×11d (FPO: [Non-Fpo])
    b86e1d54 7c8285ec 05e7e2b4 05e7e28c 05e7e2d4 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ b86e1d64)
    WARNING: Frame IP not in any known module. Following frames may be wrong.
    05e7e2d4 00000000 00000000 00000000 00000000 0×7c8285ec
    start end module name
    80800000 80a56000 nt ntkrpamp.exe Sat Feb 17 13:48:00 2007 (45D69710)
    80a56000 80a82000 hal halmacpi.dll Sat Feb 17 13:48:26 2007 (45D6972A)
    b6e94000 b6ea6720 naveng naveng.sys Fri Feb 29 09:21:51 2008 (47C75E2F)
    b6ea7000 b6f7ff80 navex15 navex15.sys Fri Feb 29 09:20:50 2008 (47C75DF2)
    b734c000 b736a000 EraserUtilDrv10741 EraserUtilDrv10741.sys Fri Jan 18 10:44:27 2008 (4790128B)
    b736a000 b73ca000 eeCtrl eeCtrl.sys Fri Jan 18 10:44:27 2008 (4790128B)
    b7492000 b74f4000 SPBBCDrv SPBBCDrv.sys Thu Jan 11 08:09:29 2007 (45A58039)
    b788e000 b78e6000 savrt savrt.sys Thu Sep 07 05:26:23 2006 (44FF3CFF)
    b790e000 b7922000 Savrtpel Savrtpel.sys Thu Sep 07 05:26:26 2006 (44FF3D02)
    b79ea000 b7a0c000 SYMEVENT SYMEVENT.SYS Wed Nov 08 11:38:37 2006 (4551513D)
    b7a0c000 b7a29000 EraserUtilDrv10710 EraserUtilDrv10710.sys Thu Feb 01 08:54:39 2007 (45C13A4F)
    b7e9a000 b7ec5000 RDPWD RDPWD.SYS Sat Feb 17 13:44:38 2007 (45D69646)
    b8235000 b8240000 TDTCP TDTCP.SYS Sat Feb 17 13:44:32 2007 (45D69640)
    b85a5000 b85bd000 mqac mqac.sys Sat Feb 17 13:44:18 2007 (45D69632)
    b85e5000 b8642000 srv srv.sys Sat Feb 17 14:27:20 2007 (45D6A048)
    b875a000 b87aa000 HTTP HTTP.sys Sat Feb 17 14:28:12 2007 (45D6A07C)
    b89b2000 b89ca000 clusnet clusnet.sys Sat Feb 17 14:02:57 2007 (45D69A91)
    b89f2000 b8a21000 RMCAST RMCAST.sys Sat Feb 17 13:57:43 2007 (45D69957)
    b9511000 b951b000 ndisuio ndisuio.sys Sat Feb 17 13:58:25 2007 (45D69981)
    b9551000 b9565000 dump_arcsas dump_arcsas.sys Thu Jul 19 08:33:04 2007 (469EB140)
    b9587000 b959c000 Cdfs Cdfs.SYS Sat Feb 17 14:27:08 2007 (45D6A03C)
    b95ce000 b95df000 Fips Fips.SYS Sat Feb 17 14:26:33 2007 (45D6A019)
    b95df000 b9655000 mrxsmb mrxsmb.sys Sat Feb 17 14:28:15 2007 (45D6A07F)
    b9655000 b9685000 rdbss rdbss.sys Sat Feb 17 14:27:37 2007 (45D6A059)
    b9685000 b96af000 afd afd.sys Sat Feb 17 14:28:16 2007 (45D6A080)
    b96af000 b96e0000 netbt netbt.sys Sat Feb 17 14:28:57 2007 (45D6A0A9)
    b9708000 b9798000 tcpip tcpip.sys Sat Feb 17 14:28:05 2007 (45D6A075)
    b9798000 b97b1000 ipsec ipsec.sys Sat Feb 17 14:29:28 2007 (45D6A0C8)
    b9821000 b9838000 ClusDisk ClusDisk.sys Sat Feb 17 14:02:55 2007 (45D69A8F)
    b9967000 b997c000 usbhub usbhub.sys Sat Feb 17 14:13:05 2007 (45D69CF1)
    b9a1d000 b9a67000 update update.sys Sat Feb 17 14:28:59 2007 (45D6A0AB)
    b9a67000 b9a9e000 rdpdr rdpdr.sys Sat Feb 17 13:51:00 2007 (45D697C4)
    b9a9e000 b9ab0000 raspptp raspptp.sys Sat Feb 17 14:29:20 2007 (45D6A0C0)
    b9ab0000 b9ac9000 ndiswan ndiswan.sys Sat Feb 17 14:29:22 2007 (45D6A0C2)
    b9ac9000 b9add000 rasl2tp rasl2tp.sys Sat Feb 17 14:29:02 2007 (45D6A0AE)
    b9add000 b9b04e00 e1000325 e1000325.sys Wed Oct 25 02:15:26 2006 (453E583E)
    b9b05000 b9b30000 b57xp32 b57xp32.sys Tue Aug 29 07:10:05 2006 (44F377CD)
    b9b30000 b9b45000 serial serial.sys Sat Feb 17 14:06:46 2007 (45D69B76)
    b9b58000 b9b7f000 ks ks.sys Sat Feb 17 14:30:40 2007 (45D6A110)
    b9b7f000 b9b93000 redbook redbook.sys Sat Feb 17 14:07:26 2007 (45D69B9E)
    b9b93000 b9ba8000 cdrom cdrom.sys Sat Feb 17 14:07:48 2007 (45D69BB4)
    b9ba8000 b9bd2000 USBPORT USBPORT.SYS Sat Feb 17 14:12:59 2007 (45D69CEB)
    b9bd2000 b9bee000 VIDEOPRT VIDEOPRT.SYS Sat Feb 17 14:10:30 2007 (45D69C56)
    b9bee000 b9d56000 ati2mtag ati2mtag.sys Thu Apr 06 11:03:52 2006 (44348518)
    ba265000 ba26c000 dxgthk dxgthk.sys Tue Mar 25 15:05:52 2003 (3E7FFFD0)
    ba26d000 ba275000 mouhid mouhid.sys Tue Mar 25 15:03:12 2003 (3E7FFF30)
    ba275000 ba27cd80 usbccgp usbccgp.sys Sat Feb 17 14:13:08 2007 (45D69CF4)
    ba27d000 ba285000 rasacd rasacd.sys Tue Mar 25 15:11:50 2003 (3E800136)
    bac95000 bac9e000 hidusb hidusb.sys Tue Mar 25 15:10:17 2003 (3E8000D9)
    bacd5000 bace0000 ptilink ptilink.sys Sat Feb 17 14:06:38 2007 (45D69B6E)
    bace5000 bacef000 Dxapi Dxapi.sys Tue Mar 25 15:06:01 2003 (3E7FFFD9)
    bad45000 bad50000 Msfs Msfs.SYS Sat Feb 17 13:50:33 2007 (45D697A9)
    bad55000 bad5e000 watchdog watchdog.sys Sat Feb 17 14:11:45 2007 (45D69CA1)
    bad75000 bad7e000 kbdhid kbdhid.sys Sat Feb 17 14:05:42 2007 (45D69B36)
    bad85000 bad92000 wanarp wanarp.sys Sat Feb 17 13:59:17 2007 (45D699B5)
    badd5000 bade0000 IPMI IPMI.sys Tue May 15 10:18:31 2007 (46491877)
    bae35000 bae44000 termdd termdd.sys Sat Feb 17 13:44:32 2007 (45D69640)
    bae65000 bae6f000 mouclass mouclass.sys Tue Mar 25 15:03:09 2003 (3E7FFF2D)
    baf0d000 baf2c000 Mup Mup.sys Sat Feb 17 14:27:41 2007 (45D6A05D)
    baf2c000 baf6b000 NDIS NDIS.sys Sat Feb 17 14:28:49 2007 (45D6A0A1)
    baf6b000 bb000000 Ntfs Ntfs.sys Sat Feb 17 14:27:23 2007 (45D6A04B)
    bf800000 bf9cf000 win32k win32k.sys Sat Feb 17 14:31:05 2007 (45D6A129)
    bf9cf000 bf9e6000 dxg dxg.sys Sat Feb 17 14:14:39 2007 (45D69D4F)
    bf9e6000 bfa29000 ati2dvag ati2dvag.dll Thu Apr 06 11:04:15 2006 (4434852F)
    bfa29000 bfa68000 ati2cqag ati2cqag.dll Thu Apr 06 09:57:18 2006 (4434757E)
    bfa68000 bfa9e000 atikvmag atikvmag.dll Thu Apr 06 10:25:29 2006 (44347C19)
    bfa9e000 bfd280c0 ati3duag ati3duag.dll Thu Apr 06 10:47:35 2006 (44348147)
    bfd29000 bfe3d180 ativvaxx ativvaxx.dll Thu Apr 06 10:40:57 2006 (44347FB9)
    f720e000 f7222000 arcsas arcsas.sys Thu Jul 19 08:33:04 2007 (469EB140)
    f7222000 f7241000 SCSIPORT SCSIPORT.SYS Sat Feb 17 14:28:41 2007 (45D6A099)
    f7241000 f725e000 atapi atapi.sys Sat Feb 17 14:07:34 2007 (45D69BA6)
    f725e000 f72be880 sddbus sddbus.sys Fri Aug 26 04:56:09 2005 (430E3069)
    f72bf000 f72e9000 volsnap volsnap.sys Sat Feb 17 14:08:23 2007 (45D69BD7)
    f72e9000 f7315000 dmio dmio.sys Sat Feb 17 14:10:44 2007 (45D69C64)
    f7315000 f733c000 ftdisk ftdisk.sys Sat Feb 17 14:08:05 2007 (45D69BC5)
    f733c000 f7352000 pci pci.sys Sat Feb 17 13:59:03 2007 (45D699A7)
    f7352000 f7386000 ACPI ACPI.sys Sat Feb 17 13:58:47 2007 (45D69997)
    f7487000 f7490000 WMILIB WMILIB.SYS Tue Mar 25 15:13:00 2003 (3E80017C)
    f7497000 f74a6000 isapnp isapnp.sys Sat Feb 17 13:58:57 2007 (45D699A1)
    f74a7000 f74b4000 PCIIDEX PCIIDEX.SYS Sat Feb 17 14:07:32 2007 (45D69BA4)
    f74b7000 f74c7000 MountMgr MountMgr.sys Sat Feb 17 14:05:35 2007 (45D69B2F)
    f74c7000 f74d2000 PartMgr PartMgr.sys Sat Feb 17 14:29:25 2007 (45D6A0C5)
    f74d7000 f74e7000 disk disk.sys Sat Feb 17 14:07:51 2007 (45D69BB7)
    f74e7000 f74f3000 Dfs Dfs.sys Sat Feb 17 13:51:17 2007 (45D697D5)
    f74f7000 f7501000 crcdisk crcdisk.sys Sat Feb 17 14:09:50 2007 (45D69C2E)
    f7577000 f7582000 TDI TDI.SYS Sat Feb 17 14:01:19 2007 (45D69A2F)
    f7587000 f7590000 raspti raspti.sys Sat Feb 17 13:59:23 2007 (45D699BB)
    f7597000 f75a0000 mssmbios mssmbios.sys Sat Feb 17 13:59:12 2007 (45D699B0)
    f75a7000 f75b6000 intelppm intelppm.sys Sat Feb 17 13:48:30 2007 (45D6972E)
    f75c7000 f75d1000 dump_diskdump dump_diskdump.sys Sat Feb 17 14:07:44 2007 (45D69BB0)
    f75d7000 f75e1000 serenum serenum.sys Sat Feb 17 14:06:44 2007 (45D69B74)
    f75e7000 f75f2000 kbdclass kbdclass.sys Sat Feb 17 14:05:39 2007 (45D69B33)
    f75f7000 f7605000 NDProxy NDProxy.SYS Sat Feb 17 13:59:21 2007 (45D699B9)
    f7607000 f7614000 netbios netbios.sys Sat Feb 17 13:58:29 2007 (45D69985)
    f7617000 f7625000 msgpc msgpc.sys Sat Feb 17 13:58:37 2007 (45D6998D)
    f7637000 f7641000 ibmhpa ibmhpa.sys Wed Aug 31 02:51:17 2005 (4314AAA5)
    f7647000 f7654000 Npfs Npfs.SYS Sat Feb 17 13:50:36 2007 (45D697AC)
    f7657000 f7665000 HIDCLASS HIDCLASS.SYS Tue Mar 25 15:10:17 2003 (3E8000D9)
    f7687000 f7690000 ndistapi ndistapi.sys Sat Feb 17 13:59:19 2007 (45D699B7)
    f76a7000 f76b6000 raspppoe raspppoe.sys Sat Feb 17 13:59:23 2007 (45D699BB)
    f76b7000 f76c3000 vga vga.sys Sat Feb 17 14:10:30 2007 (45D69C56)
    f76d7000 f76e8ac0 AACMgt AACMgt.sys Mon Nov 14 09:26:15 2005 (4377E7B7)
    f76e9000 f7707000 storport storport.sys Sat Feb 17 14:07:48 2007 (45D69BB4)
    f7707000 f770f000 kdcom kdcom.dll Tue Mar 25 15:08:00 2003 (3E800050)
    f770f000 f7717000 BOOTVID BOOTVID.dll Tue Mar 25 15:07:58 2003 (3E80004E)
    f7717000 f771e000 pciide pciide.sys Tue Mar 25 15:04:46 2003 (3E7FFF8E)
    f771f000 f7726000 dmload dmload.sys Tue Mar 25 15:08:08 2003 (3E800058)
    f775f000 f7767000 RDPCDD RDPCDD.sys Tue Mar 25 15:03:05 2003 (3E7FFF29)
    f77df000 f77e3400 usbohci usbohci.sys Sat Feb 17 14:13:01 2007 (45D69CED)
    f77e7000 f77edb80 usbehci usbehci.sys Sat Feb 17 14:12:56 2007 (45D69CE8)
    f77ef000 f77f7000 pnpmem pnpmem.sys Sat Feb 17 13:59:03 2007 (45D699A7)
    f77f7000 f77ff000 audstub audstub.sys Tue Mar 25 15:09:12 2003 (3E800098)
    f77ff000 f7807000 Fs_Rec Fs_Rec.SYS Tue Mar 25 15:08:36 2003 (3E800074)
    f7807000 f780e000 Null Null.SYS Tue Mar 25 15:03:05 2003 (3E7FFF29)
    f780f000 f7816000 Beep Beep.SYS Tue Mar 25 15:03:04 2003 (3E7FFF28)
    f7817000 f781d300 HIDPARSE HIDPARSE.SYS Sat Feb 17 14:12:35 2007 (45D69CD3)
    f781f000 f7827000 mnmdd mnmdd.SYS Tue Mar 25 15:07:53 2003 (3E800049)
    f7839000 f785f000 KSecDD KSecDD.sys Sat Feb 17 13:46:32 2007 (45D696B8)
    f785f000 f7884000 fltmgr fltmgr.sys Sat Feb 17 13:51:08 2007 (45D697CC)
    f7884000 f7897000 CLASSPNP CLASSPNP.SYS Sat Feb 17 14:28:16 2007 (45D6A080)
    f7897000 f789a280 sddfltr sddfltr.sys Fri Aug 26 04:56:09 2005 (430E3069)
    f7987000 f7988b00 sdddisk sdddisk.sys Fri Aug 26 04:56:08 2005 (430E3068)
    f7999000 f799a280 swenum swenum.sys Sat Feb 17 14:05:56 2007 (45D69B44)
    f799f000 f79a0580 USBD USBD.SYS Tue Mar 25 15:10:39 2003 (3E8000EF)
    f7b2f000 f7bdf000 ql2300 ql2300.sys Fri Nov 05 06:05:34 2004 (418AA7AE)

    Unloaded modules:
    b7160000 b7173000 naveng.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b7173000 b724c000 navex15.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b7260000 b7273000 naveng.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b7273000 b734c000 navex15.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b77ad000 b77bf000 naveng.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b77bf000 b788e000 navex15.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b7a29000 b7a8a000 eeCtrl.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b77ad000 b77bf000 naveng.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b77bf000 b788e000 navex15.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    b9b45000 b9b58000 i8042prt.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    bad35000 bad43000 imapi.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    f75b7000 f75c0000 kbdhid.sys
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    badf5000 badff000 Flpydisk.SYS
    Timestamp: unavailable (00000000)
    Checksum: 00000000
    bae05000 bae10000 Fdc.SYS
    Timestamp: unavailable (00000000)
    Checksum: 00000000

  8. molotov Says:

    michaela,

    I’d suggest enabling Verifier (http://msdn.microsoft.com/en-us/library/ms792872.aspx) on all non-MS drivers. This should get you more information when the next bugcheck happens.

  9. Crash Dump Analysis » Blog Archive » 10 Common Mistakes in Memory Analysis (Part 2) Says:

    […] would FreeHeap need a file name? See Incorrect Stack Trace pattern case study for semantic inconsistency. Why is this function on the stack […]

  10. !analyze -v : Crash Dump Analysis Patterns (Part 11) Says:

    […] Crash Dump Analysis Patterns (Part 11) […]

  11. Crash Dump Analysis » Blog Archive » Null data pointer, incorrect stack trace, changed environment, hooked functions and coincidental symbolic information: pattern cooperation Says:

    […] Unfortunately due to the lack of gui_plugin symbols the stack trace is incorrect: […]

  12. Crash Dump Analysis » Blog Archive » Inconsistent dump, blocked threads, wait chains, incorrect stack trace and process factory: pattern cooperation Says:

    […] about 45 threads are waiting that mutant 871d47d0 owned by thread 867322e0, like this one with incorrect stack trace because of inconsistent dump file […]

  13. Crash Dump Analysis » Blog Archive » Invalid pointer, incorrect stack trace, multiple exceptions, insufficient memory and memory leak: pattern cooperation Says:

    […] warning about frames suggest that stack trace could be incorrect but backwards disassembly and preceding frames show that it is correct […]

  14. Crash Dump Analysis » Blog Archive » The Importance of Symbols Says:

    […] post looks at incorrect stack traces in more detail and provides an example and explanation of why WinDbg is not able to get them right […]

  15. Crash Dump Analysis » Blog Archive » Virtualized process, incorrect stack trace, stack trace collection, multiple exceptions, optimized code and C++ exception: pattern cooperation Says:

    […] the shown above stack trace is incorrect but at the same time the first thread stack looks […]

  16. Crash Dump Analysis » Blog Archive » WOW64 process, NULL data pointer, stack overflow, main thread, incorrect stack trace, nested exceptions, hidden exception, manual dump, multiple exceptions and virtualized system: pattern cooperation Says:

    […] there is a possibility of an exception in main GUI thread and also the stack trace itself looks incorrect, suddenly sending a Windows message without any kind of a message […]

  17. Crash Dump Analysis » Blog Archive » Manual dump, virtualized process, stack trace collection, multiple exceptions, optimized code, wild code pointer, incorrect stack trace and hidden exception: pattern cooperation Says:

    […] 0×161dc2c so we might guess that this was an instance of wild code pointer or the case of incorrect stack trace. However using techniques to get exception context from hidden exceptions we get the following […]

  18. Crash Dump Analysis » Blog Archive » Reflecting on 2008 (Part 1) Says:

    […] dereferencing null debug windows crash dump analysis system_service_exception kernel32!pnlsuserinfo warning: frame ip not in any known module. following frames may be wrong. win32 error 0n2 previously announced volume is available in trade windbg script […]

  19. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 19) Says:

    […] we introduce an icon for Incorrect Stack Trace […]

  20. Crash Dump Analysis » Blog Archive » Incorrect stack trace, stack overflow, early crash dump, nested exception, problem exception handler and same vendor: pattern cooperation Says:

    […] Debugging Experts Magazine Online This case study centers on 3 process dump files (two first chance exception and one second chance exception). To recall the difference between them please read first chance exceptions explained series. When we get first and second chance exception dumps together we usually open a second chance exception dump first. However, in this case, the second chance exception dump had an incorrect stack trace: […]

  21. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 114) Says:

    […] trace may or may not be included here and it might be incorrect, heuristic and not fully discernible automatically (requires raw stack semantic analysis) like in […]

  22. Dmitry Vostokov Says:

    To check the correctness of some frames we can use the same method as described in Coincidental Symbolic Information pattern:

    http://www.dumpanalysis.org/blog/index.php/2007/08/30/crash-dump-analysis-patterns-part-24/

    We use backwards disassembly on a return address:

    0286f430 690e6daa mshtml!CBase::PrivateInvokeEx+0×6d
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0286f494 6915f5c5 jscript9!DllGetClassObject+0×18bb1

    0:005> ub 690e6daa
    jscript9!DllGetClassObject+0×18b9e:
    690e6d97 ff7514 push dword ptr [ebp+14h]
    690e6d9a ff7510 push dword ptr [ebp+10h]
    690e6d9d 8b06 mov eax,dword ptr [esi]
    690e6d9f 53 push ebx
    690e6da0 ff75ec push dword ptr [ebp-14h]
    690e6da3 ff7508 push dword ptr [ebp+8]
    690e6da6 56 push esi
    690e6da7 ff5020 call dword ptr [eax+20h]

Leave a Reply