Archive for June 26th, 2008

Opentask publishing plans

Thursday, June 26th, 2008

In July-August Opentask publisher plans to have its own website. In the mean time here is the additional list of books to be published in the next 5-7 years starting from 2009 onwards:

- Memiotics (ISBN-13: 978-1906717087)

- Voice Recognition: Command and Control (ISBN-13: 978-1906717094)

- Memory Analysis: An Interdisciplinary Approach (ISBN-13: 978-1906717117)

- Deep Down C++ (ISBN-13: 978-1906717124)

- Management Bits: An Anthology from Reductionist Manager (ISBN-13: 978-1906717131)

- Classical, Quantum and Nonlinear Memoretics (ISBN-13: 978-1906717186)

- Crash Dump: A Software Engineering Autobiography (ISBN-13: 978-1906717193)

- Memoidealism: A New Kind of Philosophy (ISBN-13: 978-1906717209)

including 10-volume edition of Software Engineering Notebooks:

- Software Engineering Notebooks, Volume 1 (ISBN-13: 978-1906717148)

Details will be announced later on the publisher’s website.

Note: the book about voice recognition stands apart from the others. This is actually the title of the first book I wanted to write 5 years ago.

- Dmitry Vostokov @ DumpAnalysis.org -

Heuristic Stack Trace in WinDbg 6.9.3.113

Thursday, June 26th, 2008

Here is another 64-bit example of Hidden Exception pattern where looking at raw stack data helps in problem identification. Opening the dump in 6.8.0004.0 version of WinDbg shows this meaningless stack trace:

00000000`00000000 ??              ???

0:035> kL
Child-SP          RetAddr           : Call Site
00000000`00000000 00000000`00000000 : 0x0

Analysis command doesn’t help too:

FAULTING_IP:
ntdll!DbgBreakPoint+0
00000000`77ef2aa0 cc              int     3

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000077ef2aa0 (ntdll!DbgBreakPoint)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0000000000000000

FAULTING_THREAD:  0000000000000e50

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT

PROCESS_NAME:  application.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

NTGLOBALFLAG:  2000000

APPLICATION_VERIFIER_FLAGS:  0

LAST_CONTROL_TRANSFER:  from 0000000000000000 to 0000000000000000

STACK_TEXT: 
00000000`00000000 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0

STACK_COMMAND:  kb

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT_STACK_CORRUPTION

FOLLOWUP_IP:
ntdll!DbgBreakPoint+0
00000000`77ef2aa0 cc              int     3

SYMBOL_NAME:  ntdll!DbgBreakPoint+0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  45d6cc72

FAILURE_BUCKET_ID:  ntdll.dll!DbgBreakPoint_80000003_STATUS_BREAKPOINT

BUCKET_ID:  X64_APPLICATION_FAULT_STATUS_BREAKPOINT_STACK_CORRUPTION_ntdll!DbgBreakPoint+0

Followup: MachineOwner
---------

However, looking at thread raw stack data allows us to get the problem stack trace showing that the full page heap enabled process detected heap corruption during free operation:

0:035> !teb
TEB at 000007fffff72000
    ExceptionList:        0000000000000000
    StackBase:            00000000080e0000
    StackLimit:           00000000080d8000

    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000007fffff72000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000918 . 0000000000000e50
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000007fffffd8000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

0:035> dds 00000000080d8000 00000000080e0000
[...]
00000000`080dd7b8  00000000`77ef3202 ntdll!KiUserExceptionDispatcher+0×52
00000000`080dd7c0  00000000`0178070a
00000000`080dd7c8  00000000`080dd7c0 ; exception context

00000000`080dd7d0  00000000`08599d30
00000000`080dd7d8  00000000`00000020
00000000`080dd7e0  00000000`00000000
00000000`080dd7e8  00000000`00000000
00000000`080dd7f0  00001fa0`0010001f
00000000`080dd7f8  0053002b`002b0033
00000000`080dd800  00000202`002b002b
00000000`080dd808  00000000`00000000
[…]

0:035> .cxr 00000000`080dd7c0
rax=0000000000000001 rbx=0000000008599d30 rcx=000077fad8cd0000
rdx=00000000ffff0165 rsi=0000000077ec0000 rdi=0000000000000000
rip=0000000077ef2aa0 rsp=00000000080ddd58 rbp=0000000000000020
 r8=00000000ffffffff  r9=0000000000000000 r10=0000000000000007
r11=0000000000000000 r12=00000000080dde70 r13=0000000077f5d300
r14=0000000077f5d2f0 r15=0000000077f86bc0
iopl=0  nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!DbgBreakPoint:
00000000`77ef2aa0 cc              int     3

0:035> kL 100
Child-SP          RetAddr           Call Site
00000000`080ddd58 00000000`77f5c78d ntdll!DbgBreakPoint
00000000`080ddd60 00000000`77f5da05 ntdll!RtlpDphReportCorruptedBlock+0×86d
00000000`080dde50 00000000`77f5a0f3 ntdll!RtlpDphNormalHeapFree+0×45
00000000`080ddfb0 00000000`77f60d5b ntdll!RtlpDebugPageHeapFree+0×203
00000000`080de0e0 00000000`77f3bcc8 ntdll!RtlDebugFreeHeap+0×3b
00000000`080de170 00000000`77edc095 ntdll!RtlFreeHeapSlowly+0×4e
00000000`080de2e0 000007ff`7fc2daab ntdll!RtlFreeHeap+0×15e
00000000`080de3f0 00000000`67fa288f msvcrt!free+0×1b
00000000`080de420 00000000`1000d3e9 dll!FreeMem+0xf

[…]
00000000`080df180 000007ff`7fe96cc9 RPCRT4!Invoke+0×65
00000000`080df1e0 000007ff`7fe9758d RPCRT4!NdrStubCall2+0×54d
00000000`080df7a0 000007ff`7fd697b4 RPCRT4!NdrServerCall2+0×1d
00000000`080df7d0 000007ff`7fde06b6 RPCRT4!DispatchToStubInCNoAvrf+0×14
00000000`080df800 000007ff`7fd6990d RPCRT4!DispatchToStubInCAvrf+0×16
00000000`080df830 000007ff`7fd69766 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×50d
00000000`080df9a0 000007ff`7fd6b214 RPCRT4!RPC_INTERFACE::DispatchToStub+0×2ec
00000000`080dfa20 000007ff`7fd6b9e3 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×63b
00000000`080dfae0 000007ff`7fd7007c RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×3bf
00000000`080dfba0 000007ff`7fd45369 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×710
00000000`080dfeb0 000007ff`7fd65996 RPCRT4!RecvLotsaCallsWrapper+0×9
00000000`080dfee0 000007ff`7fd65d51 RPCRT4!BaseCachedThreadRoutine+0xde
00000000`080dff50 00000000`77d6b6da RPCRT4!ThreadStartRoutine+0×21
00000000`080dff80 00000000`00000000 kernel32!BaseThreadStart+0×3a

Opening the dump in 6.9.3.113 version of WinDbg and running verbose analysis command shows “heuristic” stack trace (all symbols from raw stack) similar to old W2K extension stack command (see Guessing stack trace) where I highlighted exception processing hints in magenta:

0:035> !analyze -v
[...]
STACK_TEXT: 
00000000`77f15fb3 ntdll!RtlLookupFunctionTable
000007ff`5738e62c ole32!_pfnDliNotifyHook2 <PERF> (ole32+0x24e62c)
000007ff`57140000 ole32!_imp_TraceMessage <PERF> (ole32+0x0)
000007ff`57178356 ole32!ICoCreateInstanceEx
00000000`77ec0000 ntdll!_real <PERF> (ntdll+0x0)
00000000`77ef31dd ntdll!KiUserExceptionDispatcher
00000000`77f3ad68 ntdll!RtlAllocateHeapSlowly
00000000`77f967b8 ntdll!$$VProc_ImageExportDirectory
00000000`77f416ce ntdll!RtlpLookupFunctionEntryForStackWalks
00000000`77ef2aa0 ntdll!DbgBreakPoint
00000000`77ee5a36 ntdll!RtlVirtualUnwind
00000000`77f41c13 ntdll!RtlpWalkFrameChain
00000000`77f5d300 ntdll!`string'
00000000`77f5d2f0 ntdll!`string'
00000000`77f86bc0 ntdll!`string'
00000000`77ee455d ntdll!RtlpExecuteHandlerForException
00000000`77edc095 ntdll!RtlFreeHeap
00000000`77f979e4 ntdll!$$VProc_ImageExportDirectory
00000000`77ed609a ntdll!RtlCreateProcessParameters
00000000`77d5c71f kernel32!BasePushProcessParameters
00000000`77dc059b kernel32!UnhandledExceptionFilter
00000000`77ee6097 ntdll!RtlDispatchException
00000000`77f51285 ntdll!RtlpCaptureStackTraceForLogging
00000000`77f60270 ntdll!RtlDebugAllocateHeap
00000000`77f511a3 ntdll!RtlpExtendStackTraceDataBase
00000000`77d6ec00 kernel32!BasepComputeProcessPath
00000000`77d5c5b2 kernel32!BasePushProcessParameters
00000000`77d59c71 kernel32!CreateProcessInternalW
00000000`77dbc2df kernel32!BaseThreadStart
00000000`77ee6583 ntdll!_C_specific_handler
00000000`77f51432 ntdll!RtlpLogCapturedStackTrace
00000000`77f5e572 ntdll!RtlpDphLogStackTrace
00000000`77d5c4b2 kernel32!BasePushProcessParameters
00000000`77f4bb56 ntdll!DeleteNodeFromTree
00000000`77f4bf24 ntdll!RtlDeleteElementGenericTableAvl
00000000`77f574e1 ntdll!RtlpDphRemoveFromBusyList
00000000`77f5a0dd ntdll!RtlpDebugPageHeapFree
00000000`77f41799 ntdll!RtlCaptureStackBackTrace
00000000`67fa288f dll!FreeMem
00000000`77f5e559 ntdll!RtlpDphLogStackTrace
00000000`77f5a09f ntdll!RtlpDebugPageHeapFree
00000000`77f60d5b ntdll!RtlDebugFreeHeap
00000000`77f3bcc8 ntdll!RtlFreeHeapSlowly
00000000`77d4f7bc kernel32!CreateProcessInternalW
00000000`77f513bd ntdll!RtlpLogCapturedStackTrace
00000000`77ed495f ntdll!RtlDestroyProcessParameters
00000000`77d5c7c2 kernel32!BasePushProcessParameters
00000000`77dc0730 kernel32!`string’
00000000`77d813f0 kernel32!`string’
00000001`000000e0 application!_imp_RegQueryValueExW <PERF> (application+0xe0)
00000000`77ef9971 ntdll!RtlImageNtHeader
00000000`77d6b302 kernel32!BaseCreateStack
00000000`77d5c8a1 kernel32!BaseInitializeContext
00000000`77ef5a81 ntdll!CsrClientCallServer
00000000`77d5c829 kernel32!CreateProcessInternalW
00000001`00000001 application!_imp_RegQueryValueExW <PERF> (application+0×1)
00000001`00000000 application!_imp_RegQueryValueExW <PERF> (application+0×0)
000007ff`57178717 ole32!CProcessActivator::CCICallback
000007ff`571921bf ole32!CoCreateInstance
00000000`77d59620 kernel32!BaseProcessStart
00000000`77dc05d4 kernel32!UnhandledExceptionFilter
00000000`77e346e0 kernel32!__PchSym_ <PERF> (kernel32+0xf46e0)
00000000`77d6b6da kernel32!BaseThreadStart
000007ff`7fe7a934 RPCRT4!Ndr64pSizing
00000000`77f41c93 ntdll!RtlpWalkFrameChain
00000000`77edca76 ntdll!RtlAllocateHeap
00000000`77d40000 kernel32!_imp_memcpy <PERF> (kernel32+0×0)
00000000`77fa0100 ntdll!RtlpStaticDebugInfo
00000000`77ed08b3 ntdll!vsnwprintf
00000000`77dbf42c kernel32!StringCchPrintfW
00000000`77d6e314 kernel32!CloseHandle
00000000`77dc06d8 kernel32!UnhandledExceptionFilter
00000000`77e0a958 kernel32!`string’
00000000`77e29080 kernel32!CfgmgrDllString
000007ff`7fd697b4 RPCRT4!DispatchToStubInCNoAvrf
00000000`77efc2d9 ntdll!bsearch
00000000`77efc791 ntdll!RtlpFindUnicodeStringInSection
00000000`77e23454 kernel32!__PchSym_ <PERF> (kernel32+0xe3454)
00000000`77e1d324 kernel32!g_hModW03A2409
00000000`77e1d330 kernel32!g_hModW03A2409
00000000`77f39fce ntdll!RtlLookupFunctionEntry
00000000`77f39231 ntdll!RtlDispatchException
00000000`77fa3c70 ntdll!RtlpCallbackEntryList
00000000`77d92290 kernel32!_C_specific_handler
00000000`77e30033 kernel32!__PchSym_ <PERF> (kernel32+0xf0033)
000007ff`7fd65d51 RPCRT4!ThreadStartRoutine
00000000`77efc437 ntdll!RtlpLocateActivationContextSection
00000000`77ef8708 ntdll!RtlFindActivationContextSectionString
000007ff`7fc2dab0 msvcrt!free
00000000`77fc5f08 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0×105f08)
00000000`77fc5fe0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0×105fe0)
00000000`77fc5dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0×105dd0)
00000000`77fc6250 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0×106250)
00000000`77fc2614 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0×102614)
00000000`77fb2e28 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0xf2e28)
000007ff`7fc00000 msvcrt!_imp_MultiByteToWideChar <PERF> (msvcrt+0×0)
000007ff`7fc7fb78 msvcrt!bufin <PERF> (msvcrt+0×7fb78)
00000000`77ef3202 ntdll!KiUserExceptionDispatcher
00000000`77f86220 ntdll!`string’
00000000`77f5c78d ntdll!RtlpDphReportCorruptedBlock
00000000`77f5d1b0 ntdll!`string’
00000000`77f5d1e0 ntdll!`string’
00000000`77f5d200 ntdll!`string’
00000000`77f5da05 ntdll!RtlpDphNormalHeapFree
000007ff`7fde06b6 RPCRT4!DispatchToStubInCAvrf
000007ff`7fd7007c RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls
000007ff`7fd45369 RPCRT4!RecvLotsaCallsWrapper
000007ff`7fd65996 RPCRT4!BaseCachedThreadRoutine
00000000`77f57370 ntdll!RtlpDphFindBusyMemory
00000000`77f5a0f3 ntdll!RtlpDebugPageHeapFree
000007ff`57197e5b ole32!CRetailMalloc_Free
00000000`77c30000 USER32!InternalCreateDialog
000007ff`5719a21a ole32!COleStaticMutexSem::Request
00000000`77d6d6e1 kernel32!FreeLibrary
000007ff`7ebc0000 OLEAUT32!_imp_RegFlushKey <PERF> (OLEAUT32+0×0)
000007ff`56db3024 msxml3!ModelInit::~ModelInit
00000000`77d6e76c kernel32!LocalAlloc
000007ff`7fc2daab msvcrt!free
000007ff`7fd70000 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls
000007ff`7ff0b397 ADVAPI32!LocalBaseRegOpenKey
000007ff`7ff0b977 ADVAPI32!RegQueryValueExW
000007ff`7ff0b20e ADVAPI32!LocalBaseRegCloseKey
000007ff`7ff0b19f ADVAPI32!RegCloseKey
00000000`77ef7b33 ntdll!RtlNtStatusToDosError
000007ff`7fd66238 RPCRT4!LRPC_SCALL::ImpersonateClient
00000000`77efbcdf ntdll!RtlEqualSid
000007ff`7fd662a6 RPCRT4!LRPC_SCALL::RevertToSelf
000007ff`7ff0c6d4 ADVAPI32!GetTokenInformation
000007ff`7fd5cb7b RPCRT4!RpcRevertToSelf
000007ff`7fd666b2 RPCRT4!SCALL::AddToActiveContextHandles
000007ff`7fd37f76 RPCRT4!NDRSContextUnmarshall2
00000000`77f5a001 ntdll!RtlpDebugPageHeapFree
000007ff`7fd6f32b RPCRT4!DCE_BINDING::`scalar deleting destructor’
000007ff`7fd604c3 RPCRT4!RpcStringBindingParseW
000007ff`7fd30000 RPCRT4!_imp_GetSecurityDescriptorDacl <PERF> (RPCRT4+0×0)
000007ff`7fd66374 RPCRT4!NdrServerContextNewUnmarshall
000007ff`7fd605e5 RPCRT4!RpcStringFreeA
000007ff`7fd69e00 RPCRT4!NdrServerInitialize
000007ff`7fd65e81 RPCRT4!RPC_INTERFACE::CheckSecurityIfNecessary
000007ff`7fd6b9e3 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest
000007ff`7fd66c1e RPCRT4!NdrUnmarshallHandle
000007ff`7fd69a75 RPCRT4!Invoke
000007ff`7fe96cc9 RPCRT4!NdrStubCall2
00000000`77f5e500 ntdll!RtlpDphFreeDelayedBlocksFromHeap
000007ff`7fd608b4 RPCRT4!SVR_BINDING_HANDLE::SVR_BINDING_HANDLE
00000000`77ef7dbb ntdll!RtlInitializeCriticalSectionAndSpinCount
000007ff`7fd60100 RPCRT4!DCE_BINDING::StringBindingCompose
000007ff`7fe9758d RPCRT4!NdrServerCall2
000007ff`7fd5cffd RPCRT4!ParseAndCopyEndpointField
000007ff`7fd604ce RPCRT4!RpcStringBindingParseW
000007ff`7fd6990d RPCRT4!RPC_INTERFACE::DispatchToStubWorker
000007ff`7fc40b0f msvcrt!getptd
000007ff`7fd37eaf RPCRT4!RpcServerInqCallAttributesW
000007ff`7fd65e9c RPCRT4!RPC_INTERFACE::CheckSecurityIfNecessary
000007ff`7fd69766 RPCRT4!RPC_INTERFACE::DispatchToStub
000007ff`7fd6b214 RPCRT4!LRPC_SCALL::DealWithRequestMessage
000007ff`7fd70466 RPCRT4!LRPC_ADDRESS::DereferenceAssociation
000007ff`7fd6ee28 RPCRT4!LRPC_SASSOCIATION::DealWithCopyMessage
000007ff`7fd65d30 RPCRT4!ThreadStartRoutine
00000000`77d6b6a0 kernel32!BaseThreadStart

- Dmitry Vostokov @ DumpAnalysis.org -