Software Diagnostics Library » Crash Dump Analysis

Archive for the ‘Crash Dump Analysis’ Category

Crash Dump Analysis Patterns (Part 25d)

Tuesday, October 21st, 2014

Some troubleshooting and debugging techniques involve saving every Stack Trace that leads to a specific action such as a memory allocation of opening of a resource handle to be saved in some region in memory, called stack trace database. Typical pattern usage examples include Process Heap Memory Leak, Insufficient Memory due to Handle Leak. Typical entry in such a database consists of return addresses saved during function calls (which may be Truncated Stack Trace):

00000000`00325da0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325da8 00000001`3fd72239 AllocFree!_ioinit+0×2cd 00000000`00325db0 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5 00000000`00325db8 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325dc0 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d

0:001> ub 00000001`3fd72239 AllocFree!_ioinit+0×2af: 00000001`3fd7221b cmp eax,3 00000001`3fd7221e jne AllocFree!_ioinit+0×2be (00000001`3fd7222a) 00000001`3fd72220 movsx eax,byte ptr [rbx+8] 00000001`3fd72224 or eax,8 00000001`3fd72227 mov byte ptr [rbx+8],al 00000001`3fd7222a lea rcx,[rbx+10h] 00000001`3fd7222e mov edx,0FA0h 00000001`3fd72233 call qword ptr [AllocFree!_imp_InitializeCriticalSectionAndSpinCount (00000001`3fd78090)

This slightly differs from ‘k’-style stack trace format where the return address belongs to the function on the next line if moving downwards:

0:000> k Child-SP RetAddr Call Site 00000000`002ff9f8 000007fe`fd5e1203 ntdll!ZwDelayExecution+0xa 00000000`002ffa00 00000001`3fd71018 KERNELBASE!SleepEx+0xab 00000000`002ffaa0 00000001`3fd71194 AllocFree!wmain+0×18 00000000`002ffad0 00000000`773759ed AllocFree!__tmainCRTStartup+0×144 00000000`002ffb10 00000000`774ac541 kernel32!BaseThreadInitThunk+0xd 00000000`002ffb40 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub 00000001`3fd71194 AllocFree!__tmainCRTStartup+0×11b: 00000001`3fd7116b je AllocFree!__tmainCRTStartup+0×124 (00000001`3fd71174) 00000001`3fd7116d mov ecx,eax 00000001`3fd7116f call AllocFree!_amsg_exit (00000001`3fd718ec) 00000001`3fd71174 mov r8,qword ptr [AllocFree!_wenviron (00000001`3fd80868)] 00000001`3fd7117b mov qword ptr [AllocFree!__winitenv (00000001`3fd80890)],r8 00000001`3fd71182 mov rdx,qword ptr [AllocFree!__wargv (00000001`3fd80858)] 00000001`3fd71189 mov ecx,dword ptr [AllocFree!__argc (00000001`3fd8084c)] 00000001`3fd7118f call AllocFree!wmain (00000001`3fd71000)

Sometimes we can see such traces as Execution Residue inside a stack or some other region. If user mode stack trace database is enabled in gflags.exe we might be able to dump the specific database region:

0:001> !gflag Current NtGlobalFlag contents: 0x00001000 ust - Create user mode stack trace database

0:001> !address [...] BaseAddress EndAddress+1 RegionSize Type State Protect Usage ------------------------------------------------------------------------------------------------------------------------ [...] + 0`00300000 0`00326000 0`00026000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database] 0`00326000 0`01aff000 0`017d9000 MEM_PRIVATE MEM_RESERVE Other [Stack Trace Database] 0`01aff000 0`01b00000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Other [Stack Trace Database] […]

0:001> dps 0`00326000-1000 0`00326000 […] 00000000`003257e0 00000000`00000000 00000000`003257e8 00030001`00001801 00000000`003257f0 00000000`774c34eb ntdll!LdrpInitializeProcess+0×7e6 00000000`003257f8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325800 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325808 00000000`00000000 00000000`00325810 00000000`00000000 00000000`00325818 00030002`00001801 00000000`00325820 00000000`774c3511 ntdll!LdrpInitializeProcess+0×80c 00000000`00325828 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325830 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325838 00000000`00000000 00000000`00325840 00000000`00000000 00000000`00325848 00040003`00001801 00000000`00325850 00000000`774bda86 ntdll!RtlCreateHeap+0×506 00000000`00325858 00000000`774c3557 ntdll!LdrpInitializeProcess+0×851 00000000`00325860 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325868 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325870 00000000`00000000 00000000`00325878 00050004`00002801 00000000`00325880 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a 00000000`00325888 00000000`774bdaee ntdll!RtlCreateHeap+0×56e 00000000`00325890 00000000`774c3557 ntdll!LdrpInitializeProcess+0×851 00000000`00325898 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`003258a0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`003258a8 00000000`00000000 00000000`003258b0 00000000`00000000 00000000`003258b8 00030005`00001801 00000000`003258c0 00000000`774c359e ntdll!LdrpInitializeProcess+0×902 00000000`003258c8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`003258d0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`003258d8 00000000`00000000 00000000`003258e0 00000000`00000000 00000000`003258e8 00030006`00001801 00000000`003258f0 00000000`774c35af ntdll!LdrpInitializeProcess+0×913 00000000`003258f8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325900 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325908 00000000`00000000 00000000`00325910 00000000`00000000 00000000`00325918 00090007`00004801 00000000`00325920 00000000`774bda86 ntdll!RtlCreateHeap+0×506 00000000`00325928 00000000`774c47ff ntdll!CsrpConnectToServer+0×41f 00000000`00325930 00000000`774c43c5 ntdll!CsrClientConnectToServer+0×230 00000000`00325938 000007fe`fd5ee232 KERNELBASE!KernelBaseDllInitialize+0×148 00000000`00325940 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325948 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325950 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325958 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325960 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325968 00000000`00000000 00000000`00325970 00000000`00000000 00000000`00325978 000a0008`00004801 00000000`00325980 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a 00000000`00325988 00000000`774bdaee ntdll!RtlCreateHeap+0×56e 00000000`00325990 00000000`774c47ff ntdll!CsrpConnectToServer+0×41f 00000000`00325998 00000000`774c43c5 ntdll!CsrClientConnectToServer+0×230 00000000`003259a0 000007fe`fd5ee232 KERNELBASE!KernelBaseDllInitialize+0×148 00000000`003259a8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`003259b0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`003259b8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`003259c0 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`003259c8 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`003259d0 00000000`00000000 00000000`003259d8 00080009`00003801 00000000`003259e0 000007fe`fd5edf81 KERNELBASE!NlsProcessInitialize+0×11 00000000`003259e8 000007fe`fd604439 KERNELBASE!BaseNlsDllInitialize+0×29 00000000`003259f0 000007fe`fd5ee446 KERNELBASE!KernelBaseDllInitialize+0×40c 00000000`003259f8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325a00 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325a08 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325a10 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325a18 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325a20 00000000`00000000 00000000`00325a28 0008000a`00003801 00000000`00325a30 000007fe`fd5edfa0 KERNELBASE!NlsProcessInitialize+0×30 00000000`00325a38 000007fe`fd604439 KERNELBASE!BaseNlsDllInitialize+0×29 00000000`00325a40 000007fe`fd5ee446 KERNELBASE!KernelBaseDllInitialize+0×40c 00000000`00325a48 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325a50 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325a58 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325a60 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325a68 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325a70 00000000`00000000 00000000`00325a78 0007000b`00003801 00000000`00325a80 000007fe`fd604a21 KERNELBASE!BasepInitComputerNameCache+0×11 00000000`00325a88 000007fe`fd603d20 KERNELBASE!KernelBaseDllInitialize+0×419 00000000`00325a90 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325a98 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325aa0 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325aa8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325ab0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325ab8 00000000`00000000 00000000`00325ac0 00000000`00000000 00000000`00325ac8 0006000c`00002801 00000000`00325ad0 00000000`77375699 kernel32!BaseDllInitialize+0×2f9 00000000`00325ad8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325ae0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325ae8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325af0 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325af8 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325b00 00000000`00000000 00000000`00325b08 0007000d`00003801 00000000`00325b10 00000000`773771f7 kernel32!InitializeConsoleConnectionInfo+0xe7 00000000`00325b18 00000000`773756ae kernel32!BaseDllInitialize+0×30e 00000000`00325b20 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325b28 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325b30 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325b38 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325b40 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325b48 00000000`00000000 00000000`00325b50 00000000`00000000 00000000`00325b58 0009000e`00004801 00000000`00325b60 00000000`774bda86 ntdll!RtlCreateHeap+0×506 00000000`00325b68 00000000`773787f7 kernel32!ConsoleConnect+0×1d7 00000000`00325b70 00000000`773770de kernel32!ConnectConsoleInternal+0×147 00000000`00325b78 00000000`773756fe kernel32!BaseDllInitialize+0×35e 00000000`00325b80 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325b88 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325b90 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325b98 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325ba0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325ba8 00000000`00000000 00000000`00325bb0 00000000`00000000 00000000`00325bb8 000a000f`00004801 00000000`00325bc0 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a 00000000`00325bc8 00000000`774bdaee ntdll!RtlCreateHeap+0×56e 00000000`00325bd0 00000000`773787f7 kernel32!ConsoleConnect+0×1d7 00000000`00325bd8 00000000`773770de kernel32!ConnectConsoleInternal+0×147 00000000`00325be0 00000000`773756fe kernel32!BaseDllInitialize+0×35e 00000000`00325be8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325bf0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325bf8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325c00 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325c08 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325c10 00000000`00000000 00000000`00325c18 00060010`00002801 00000000`00325c20 00000000`773757dc kernel32!BaseDllInitialize+0×43c 00000000`00325c28 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325c30 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325c38 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325c40 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325c48 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325c50 00000000`00000000 00000000`00325c58 00060011`00002801 00000000`00325c60 00000000`7737582c kernel32!BaseDllInitialize+0×48c 00000000`00325c68 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe 00000000`00325c70 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa 00000000`00325c78 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b 00000000`00325c80 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0 00000000`00325c88 00000000`774ac34e ntdll!LdrInitializeThunk+0xe 00000000`00325c90 00000000`00000000 00000000`00325c98 00060012`0000280e 00000000`00325ca0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325ca8 00000001`3fd7319f AllocFree!_mtinitlocks+0×43 00000000`00325cb0 00000001`3fd717fc AllocFree!_mtinit+0×10 00000000`00325cb8 00000001`3fd710e4 AllocFree!__tmainCRTStartup+0×94 00000000`00325cc0 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325cc8 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325cd0 00000000`00000000 00000000`00325cd8 000b0013`00005801 00000000`00325ce0 00000000`774c1131 ntdll!RtlpActivateLowFragmentationHeap+0×181 00000000`00325ce8 00000000`774c0f97 ntdll!RtlpPerformHeapMaintenance+0×27 00000000`00325cf0 00000000`774c0f5b ntdll!RtlpAllocateHeap+0×1819 00000000`00325cf8 00000000`774d34d8 ntdll!RtlAllocateHeap+0×16c 00000000`00325d00 00000000`774a9300 ntdll!RtlInitializeCriticalSectionAndSpinCount+0×183 00000000`00325d08 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325d10 00000001`3fd7319f AllocFree!_mtinitlocks+0×43 00000000`00325d18 00000001`3fd717fc AllocFree!_mtinit+0×10 00000000`00325d20 00000001`3fd710e4 AllocFree!__tmainCRTStartup+0×94 00000000`00325d28 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325d30 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325d38 00000000`00000000 00000000`00325d40 00000000`00000000 00000000`00325d48 00070014`00003801 00000000`00325d50 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325d58 00000001`3fd7312f AllocFree!_mtinitlocknum+0×8f 00000000`00325d60 00000001`3fd72ff7 AllocFree!_lock+0×23 00000000`00325d68 00000001`3fd71f9b AllocFree!_ioinit+0×2f 00000000`00325d70 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5 00000000`00325d78 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325d80 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325d88 00000000`00000000 00000000`00325d90 00000000`00000000 00000000`00325d98 00050015`00002803 00000000`00325da0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa 00000000`00325da8 00000001`3fd72239 AllocFree!_ioinit+0×2cd 00000000`00325db0 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5 00000000`00325db8 00000000`773759ed kernel32!BaseThreadInitThunk+0xd 00000000`00325dc0 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d 00000000`00325dc8 00000000`00000000 00000000`00325dd0 00000000`00000000 […]

This database corresponds to this simple program:

int _tmain(int argc, _TCHAR* argv[])
{
    free(malloc(256));
    Sleep(-1);
    return 0;
}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 25c)

Sunday, October 12th, 2014

Sometimes threads related to file system operations may be blocked with not easily recognizable 3rd-party Top Module with only OS vendor modules such as NTFS or fltmgr present:

nt!KiSwapContext+0x7a nt!KiCommitThreadWait+0x1d2 nt!KeWaitForSingleObject+0x19f nt!FsRtlCancellableWaitForMultipleObjects+0x5e nt!FsRtlCancellableWaitForSingleObject+0x27 fltmgr! ?? ::FNODOBFM::`string’+0×2bfa fltmgr!FltpCreate+0×2a9 nt!IopParseDevice+0×14d3 nt!ObpLookupObjectName+0×588 nt!ObOpenObjectByName+0×306 nt!IopCreateFile+0×2bc nt!NtCreateFile+0×78 nt!KiSystemServiceCopyEnd+0×13 ntdll!NtCreateFile+0xa […]

We see the same modules in I/O Request Stack Trace from the thread IRP. But because we see filter manager involved there may be some 3rd-party file system filters involved. Such filters are called before a device processes a request and also upon the completion of the request. There may be different filter callbacks registered for each case and they form a similar structure like I/O stack locations (we call this pattern Filter Stack Trace):

If one of such filters is blocked in a wait chain this may not be visible on I/O request or thread stacks because of possible asynchronous processing. But we may use !fltkd.irpctrl debugging extension command to examine the IRP context:

0: kd> !irp fffffa80162aa230 cmd flg cl Device File Completion-Context [...] [ 0, 0] 0 0 fffffa800cb28030 00000000 fffff880012048f0-fffffa8016f64010 \FileSystem\Ntfs fltmgr!FltpSynchronizedOperationCompletion Args: 00000000 00000000 00000000 00000000 >[ 0, 0] 0 1 fffffa800ca00890 fffffa801060d070 00000000-00000000 pending \FileSystem\FltMgr Args: fffff88014450868 02000060 00000006 00000000

0: kd> !fltkd.irpctrl fffffa8016f64010 [...] Cmd IrpFl OpFl CmpFl Instance FileObjt Completion-Context Node Adr --------- -------- ----- ----- -------- -------- ------------------ -------- [0,0] 00000884 00 0000 fffffa800d29c010 fffffa801060d070 fffff8800518b474-0000000000000000 fffffa8016f641e0 ("luafv","luafv") luafv!LuafvPostCreate Args: fffff88014450868 0000000002000060 0000000000000006 0000000000000000 0000000000000000 0000000000000000 >[0,0] 00000884 00 0000 fffffa800e8051d0 fffffa801060d070 fffff88006808440-0000000000000000 fffffa8016f64160 (”3rdPartyFilter”,”3rdPartyFilter Instance”) FilterA!FltDriver_PostOperationCallback Args: fffff88014450868 0000000002000060 0000000000000006 0000000000000000 0000000000000000 0000000000000000 […]

So we see that FilterA module may be involved in blocking the thread (Blocking Module pattern extended to I/O request and filter stack traces).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 25b)

Sunday, October 12th, 2014

If a thread has an associated I/O Request Packet (IRP) we may see another type of a stack trace we call I/O Request Stack Trace. It also grows bottom-up as can be seen on the diagram 3. We can see this stack trace by using !irp WinDbg command:

0: kd> !thread fffffa801827a4c0 3f THREAD fffffa801827a4c0 Cid 06c0.50cc Teb: 000007ffffec8000 Win32Thread: fffff900c1c64010 WAIT: (Executive) KernelMode Alertable fffffa8016f64028 SynchronizationEvent IRP List: fffffa80162aa230: (0006,03a0) Flags: 00000884 Mdl: 00000000 […] nt!KiSwapContext+0×7a nt!KiCommitThreadWait+0×1d2 nt!KeWaitForSingleObject+0×19f nt!FsRtlCancellableWaitForMultipleObjects+0×5e nt!FsRtlCancellableWaitForSingleObject+0×27 fltmgr! ?? ::FNODOBFM::`string’+0×2bfa fltmgr!FltpCreate+0×2a9 nt!IopParseDevice+0×14d3 nt!ObpLookupObjectName+0×588 nt!ObOpenObjectByName+0×306 nt!IopCreateFile+0×2bc nt!NtCreateFile+0×78 nt!KiSystemServiceCopyEnd+0×13 ntdll!NtCreateFile+0xa […]

0: kd> !irp fffffa80162aa230 Irp is active with 10 stacks 10 is current (= 0xfffffa80162aa588) No Mdl: No System Buffer: Thread fffffa801827a4c0: Irp stack trace. cmd flg cl Device File Completion-Context [ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 fffffa800cb28030 00000000 fffff880012048f0-fffffa8016f64010 \FileSystem\Ntfs fltmgr!FltpSynchronizedOperationCompletion Args: 00000000 00000000 00000000 00000000 >[ 0, 0] 0 1 fffffa800ca00890 fffffa801060d070 00000000-00000000 pending \FileSystem\FltMgr Args: fffff88014450868 02000060 00000006 00000000

We see the current stack trace pointer points to the bottom I/O stack location. Non-empty top locations are analogous to Past Stack Trace. Further exploration of Device and File column information may point to further troubleshooting directions such as the Blocking File pattern example.

By analogy with Stack Trace Collection pattern that dumps stack traces from all threads based on memory dump type there is also I/O Stack Trace Collection pattern that dumps I/O request stack traces from all IRPs that were possible to find.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 214)

Wednesday, October 8th, 2014

When we look at a stack trace in a memory dump we see only the current thread execution snapshot of function calls. Consider this stack trace, for example, from Spiking Thread:

0:000> k Child-SP RetAddr Call Site 00000000`0012d010 00000000`76eb59ed App!WinMain+0x1eda 00000000`0012f7c0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd 00000000`0012f7f0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

By looking at Rough Stack Trace we may be able to reconstruct Past Stack Trace of what had happened just before the memory snapshot was taken:

0:000> k Child-SP RetAddr Call Site 00000000`0012cfd8 00000000`76fd9e9e user32!ZwUserGetMessage+0xa 00000000`0012cfe0 00000000`ffd91a8c user32!GetMessageW+0x34 00000000`0012d010 00000000`76eb59ed App!WinMain+0x1dca 00000000`0012f7c0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd 00000000`0012f7f0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

The stack region “time” zones are illustrated on the following picture:

The “Future” zone takes its name from the not yet executed returns. Of course, each stack subtrace generates its own partition. A similar version of this pattern was first introduced in Debugging TV Frames episode 0×24. You watch the video here and can find source code, WinDbg logs, and presentation here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 213)

Tuesday, October 7th, 2014

Rough Stack Trace is an example of more general Execution Residue pattern or Caller-n-Callee for managed space. It’s just a collection of symbolic references (may also include Coincidental Symbolic Information) from the thread stack region or its fragment. In WinDbg we can get it by using dpS command:

0:003> !teb TEB at 000007fffffd6000 ExceptionList: 0000000000000000 StackBase: 0000000002450000 StackLimit: 000000000244b000 SubSystemTib: 0000000000000000 FiberData: 0000000000001e00 ArbitraryUserPointer: 0000000000000000 Self: 000007fffffd6000 EnvironmentPointer: 0000000000000000 ClientId: 00000000000047fc . 0000000000004824 RpcHandle: 0000000000000000 Tls Storage: 000007fffffd6058 PEB Address: 000007fffffda000 LastErrorValue: 0 LastStatusValue: c0000302 Count Owned Locks: 0 HardErrorMode: 0

0:003> dpS 000000000244b000 0000000002450000 000007fe`fd4a8a2e ole32!InternalVerifyStackAvailable+0x44 [d:\winmain\minio\safealloca\alloca.c @ 317] 000007fe`fd4a8a2e ole32!InternalVerifyStackAvailable+0x44 [d:\winmain\minio\safealloca\alloca.c @ 317] 000007fe`fd4a8a2e ole32!InternalVerifyStackAvailable+0x44 [d:\winmain\minio\safealloca\alloca.c @ 317] 00000000`771d5430 ntdll!RtlpInterceptorRoutines 00000000`771134d8 ntdll!RtlAllocateHeap+0x16c 00000000`770ec9c3 ntdll!RtlAppendUnicodeStringToString+0x53 00000000`76eaebe5 kernel32!Wow64RedirectKeyPathInternal+0x2b7 00000000`770ec9c3 ntdll!RtlAppendUnicodeStringToString+0x53 00000000`771140fd ntdll!RtlFreeHeap+0x1a6 00000000`76eaec01 kernel32!ConstructKernelKeyPath+0x15f 00000000`76eaedd3 kernel32!Wow64NtOpenKey+0xee 00000000`771140fd ntdll!RtlFreeHeap+0x1a6 00000000`76ebc8aa kernel32!BaseRegOpenClassKeyFromLocation+0x3ba 00000000`76f3edf0 kernel32!`string' 00000000`771d5430 ntdll!RtlpInterceptorRoutines 00000000`76ebc9b9 kernel32!BaseRegGetUserPrefixLength+0xea 00000000`76f3ee38 kernel32!`string' 00000000`76f3edc8 kernel32!`string' 00000000`76ebc3a8 kernel32!BaseRegGetKeySemantics+0x1b8 00000000`771150d3 ntdll!RtlNtStatusToDosError+0x27 00000000`76eb36b7 kernel32!LocalBaseRegOpenKey+0x276 000007fe`fd4b6c79 ole32!GetUnquotedPath+0x29 [d:\w7rtm\com\ole32\com\objact\dllcache.cxx @ 2256] 000007fe`fd4b7019 ole32!CClassCache::CDllPathEntry::NegotiateDllInstantiationProperties2+0x145 [d:\w7rtm\com\ole32\com\objact\dllcache.cxx @ 3092] 00000000`771d5430 ntdll!RtlpInterceptorRoutines 00000000`771134d8 ntdll!RtlAllocateHeap+0x16c 00000000`77115cc4 ntdll!RtlpAllocateHeap+0xc12 000007fe`fdc10359 usp10!CUspShapingClient::AllocMem+0x49 000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12 000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12 000007fe`fdc1d4f1 usp10!UspFreeMem+0x61 000007fe`fdc4896e usp10!COtlsClient::FreeMem+0xe 000007fe`fdc6e817 usp10!ApplyFeatures+0xa17 000007fe`fdc6f2f2 usp10!ApplyLookup+0x592 000007fe`fdc48901 usp10!COtlsClient::GetDefaultGlyphs+0x131 000007fe`fdc60100 usp10!HangulEngineGetGlyphs+0x2c0 000007fe`fdc10359 usp10!CUspShapingClient::AllocMem+0x49 000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12 000007fe`fdc10359 usp10!CUspShapingClient::AllocMem+0x49 000007fe`fdc1d4f1 usp10!UspFreeMem+0x61 000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12 000007fe`fdc1d4f1 usp10!UspFreeMem+0x61 000007fe`fdc4896e usp10!COtlsClient::FreeMem+0xe 000007fe`fdc6e817 usp10!ApplyFeatures+0xa17 000007fe`fdc6aaa8 usp10!RePositionOtlGlyphs+0x238 000007fe`fdc48901 usp10!COtlsClient::GetDefaultGlyphs+0x131 000007fe`fdc60100 usp10!HangulEngineGetGlyphs+0x2c0 000007fe`fdc48798 usp10!COtlsClient::ReleaseOtlTable+0x78 000007fe`fdc6ae85 usp10!otlResourceMgr::detach+0xc5 00000000`7717c63e ntdll!EtwEventWriteNoRegistration+0xae 000007fe`fdc48a99 usp10!COtlsClient::Release+0x49 00000000`771150d3 ntdll!RtlNtStatusToDosError+0x27 00000000`7716bd85 ntdll!WaitForWerSvc+0x85 00000000`7717b94e ntdll!WerpAllocateAndInitializeSid+0xbe 00000000`7716bd90 ntdll! ?? ::FNODOBFM::`string' 00000000`77175dcf ntdll!WerpFreeSid+0x3f 00000000`7718123d ntdll!SendMessageToWERService+0x22d 00000000`77181260 ntdll! ?? ::FNODOBFM::`string' 00000000`77182308 ntdll!ReportExceptionInternal+0xc8 000007fe`fd061430 KERNELBASE!WaitForMultipleObjectsEx+0xe8 00000000`76ec1723 kernel32!WaitForMultipleObjectsExImplementation+0xb3 00000000`76f3b5e5 kernel32!WerpReportFaultInternal+0x215 00000000`76f3b767 kernel32!WerpReportFault+0x77 00000000`76f3b7bf kernel32!BasepReportFault+0x1f 00000000`76f3b9dc kernel32!UnhandledExceptionFilter+0x1fc 00000000`77118d7e ntdll!RtlpFindUnicodeStringInSection+0x50e 00000000`771198fc ntdll!LdrpFindLoadedDll+0x10c 00000000`770e9caa ntdll!RtlDecodePointer+0x2a 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`771e8180 ntdll!`string'+0xc040 00000000`771e818c ntdll!`string'+0xc04c 00000000`77153398 ntdll! ?? ::FNODOBFM::`string'+0x2365 00000000`770d85c8 ntdll!_C_specific_handler+0x8c 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`770ec541 ntdll!RtlUserThreadStart+0x1d 00000000`770e9d2d ntdll!RtlpExecuteHandlerForException+0xd 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`770d91cf ntdll!RtlDispatchException+0x45a 00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0) 00000000`7711920a ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3da 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`771e8180 ntdll!`string'+0xc040 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`770ec541 ntdll!RtlUserThreadStart+0x1d 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`771d7718 ntdll!LdrpDefaultExtension 00000000`770d852c ntdll!_C_specific_handler 00000000`771e8180 ntdll!`string'+0xc040 000007fe`ff3625c0 msctf!s_szCompClassName 000007fe`fd602790 ole32!`string' 00000000`770e7a33 ntdll!LdrpFindOrMapDll+0x138 00000000`771192a8 ntdll!LdrpApplyFileNameRedirection+0x2d3 000007fe`fd602848 ole32!`string' 00000000`771d5430 ntdll!RtlpInterceptorRoutines 00000000`77113448 ntdll!RtlAllocateHeap+0xe4 00000000`76fd88b8 user32!GetPropW+0x4d 00000000`76fd88b8 user32!GetPropW+0x4d 00000000`76fd7931 user32!IsWindow+0x9 00000000`76fd7931 user32!IsWindow+0x9 00000000`770f41c8 ntdll!RtlpReAllocateHeap+0x178 000007fe`fb601381 uxtheme!CThemeWnd::_PreDefWindowProc+0x31 00000000`76eb59e0 kernel32!BaseThreadInitThunk 00000000`ffdbdb32 calc!CTimedCalc::Start+0xa9 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0) 00000000`ffe0ac64 calc!_dyn_tls_init_callback <PERF> (calc+0x7ac64) 00000000`76ea0000 kernel32!TestResourceDataMatchEntry <PERF> (kernel32+0x0) 00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0) 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`76fd760e user32!RealDefWindowProcW+0x5a 000007fe`fb600037 uxtheme!operator delete <PERF> (uxtheme+0x37) 00000000`77111248 ntdll!KiUserExceptionDispatch+0x2e 000007fe`fb63fb40 uxtheme!$$VProc_ImageExportDirectory 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2 00000000`76fe76c2 user32!DefDlgProcW+0x36 00000000`76fd9bef user32!UserCallWinProcCheckWow+0x1cb 00000000`76fd9b43 user32!UserCallWinProcCheckWow+0x99 00000000`76fd9bef user32!UserCallWinProcCheckWow+0x1cb 00000000`76fd72cb user32!DispatchClientMessage+0xc3 00000000`770e46b4 ntdll!NtdllDialogWndProc_W 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2 00000000`77101530 ntdll!NtdllDispatchMessage_W 00000000`76fe505b user32!DialogBox2+0x2ec 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0) 00000000`76fe4edd user32!InternalDialogBox+0x135 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0) 00000000`76fe4f52 user32!DialogBoxIndirectParamAorW+0x58 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0) 00000000`76fdd476 user32!DialogBoxParamW+0x66 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`ffdbdafa calc!CTimedCalc::WatchDogThread+0x72 00000000`76eb59ed kernel32!BaseThreadInitThunk+0xd 00000000`770ec541 ntdll!RtlUserThreadStart+0x1d 00000000`76f3b7e0 kernel32!UnhandledExceptionFilter 00000000`76f3b7e0 kernel32!UnhandledExceptionFilter

The name for this pattern comes from rough sets in mathematics.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 212)

Sunday, October 5th, 2014

Although in the case of system hangs we, usually, recommend dumping Stack Trace Collection, in some cases it is very time-consuming, especially when it involves thousands of processes such as in modern terminal services environments. In such a case, if the problem description indicates the last action such as a not progressing user logon or a recently launched process we first check the tail of the corresponding linked list where Last Object is usually added to the tail of the list:

Sometimes we can simply check the end of some enumerated collection such as sessions (dotted lines represent ALPC Wait Chains):

This analysis pattern can be added to the first tier of RSDP. If nothing found around a couple of Last Objects we then resort to the analysis of entire linked lists.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | 1 Comment »

Crash Dump Analysis Patterns (Part 211)

Saturday, October 4th, 2014

In process heap Memory Leak pattern we recommended acquiring sequential memory dumps spaced by 100MB. Unfortunately customers may send memory dumps spaced more closely, say by 10 - 20 MB or less after memory consumption growth already started some time in the past, for example, when they feel further process growth may impact their system performance. The analysis of process heap from memory dumps with enabled user mode stack database and corresponding UMDH log differences may show only Memory Fluctuation, where memory increases for specific stack trace allocations may follow by decreases or by small increases (S_i is for memory dump size [horizontal bars], t_i is for memory acquisition time):

In such cases it is difficult to choose among various local memory fluctuations to continue further investigation. However, a baseline process memory dump, for example, just after process start, helps to choose which stack trace allocations investigate first: those having bigger absolute memory allocation increase (Allocation Stack Trace B):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | 1 Comment »

Crash Dump Analysis Patterns (Part 210)

Monday, September 8th, 2014

Here we provide another variant of a general Wait Chain pattern related to RtlAcquireResourceShared and RtlAcquireResourceExclusive calls:

THREAD fffffa8052d66060 Cid 03c0.3240 Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable fffffa804a79ad50 Semaphore Limit 0x7fffffff Impersonation token: fffff8a01b19d060 (Level Impersonation) DeviceMap fffff8a0035276c0 Owning Process fffffa804a16b260 Image: lsm.exe Attached Process N/A Image: N/A Wait Start TickCount 73343513 Ticks: 1460259 (0:06:20:16.546) Context Switch Count 17 IdealProcessor: 1 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address ntdll!TppWorkerThread (0x000000007735fbf0) Stack Init fffff8800e870db0 Current fffff8800e870900 Base fffff8800e871000 Limit fffff8800e86b000 Call 0 Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Kernel stack not resident. Child-SP RetAddr Call Site fffff880`0e870940 fffff800`01c76972 nt!KiSwapContext+0x7a fffff880`0e870a80 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2 fffff880`0e870b10 fffff800`01f7b2be nt!KeWaitForSingleObject+0x19f fffff880`0e870bb0 fffff800`01c801d3 nt!NtWaitForSingleObject+0xde fffff880`0e870c20 00000000`773912fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0e870c20) 00000000`022ae6c8 00000000`773470b4 ntdll!NtWaitForSingleObject+0xa 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0 00000000`022ae710 00000000`ff401675 lsm!CAutoSharedLock::CAutoSharedLock+0×61 00000000`022ae7e0 00000000`ff402c68 lsm!CTSSession::getTerminal+0×21 00000000`022ae820 000007fe`fd8bff85 lsm!RpcGetEnumResult+0×202 00000000`022ae980 000007fe`fd8b4de2 RPCRT4!Invoke+0×65 00000000`022ae9e0 000007fe`fd8b17bd RPCRT4!NdrStubCall2+0×32a 00000000`022af000 000007fe`fd8b3254 RPCRT4!NdrServerCall2+0×1d 00000000`022af030 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0×14 00000000`022af060 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146 00000000`022af180 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0×149 00000000`022af260 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0×20d 00000000`022af390 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf 00000000`022af4d0 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5 00000000`022af560 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0×26b 00000000`022af5f0 00000000`7713652d ntdll!TppWorkerThread+0×3f8 00000000`022af8f0 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd 00000000`022af920 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

These functions are undocumented but ReactOS source code shows they all take a pointer to RTL_RESOURCE structure which has handles to a shared and exclusive semaphores:

RTL_CRITICAL_SECTION Lock HANDLE SharedSemaphore ULONG SharedWaiters HANDLE ExclusiveSemaphore ULONG ExclusiveWaiters LONG NumberActive HANDLE OwningThread ULONG TimeoutBoost PVOID DebugInfo

To double check that we disassemble RtlAcquireResourceShared and check the return address from NtWaitForSingleObject call (00000000`773470b4):

0: kd> .thread /r /p fffffa8052d66060 Implicit thread is now fffffa80`52d66060 Implicit process is now fffffa80`4a16b260 Loading User Symbols ..........................................

0: kd> uf ntdll!RtlAcquireResourceShared [...] ntdll!RtlAcquireResourceShared+0xc2: 00000000`773470a6 488b4b28 mov rcx,qword ptr [rbx+28h] 00000000`773470aa 4c8bc6 mov r8,rsi 00000000`773470ad 33d2 xor edx,edx 00000000`773470af e83ca20400 call ntdll!NtWaitForSingleObject (00000000`773912f0) 00000000`773470b4 3d02010000 cmp eax,102h 00000000`773470b9 0f8402800600 je ntdll! ?? ::FNODOBFM::`string’+0×12629 (00000000`773af0c1) […] ntdll!RtlAcquireResourceShared: 00000000`77352af0 48895c2420 mov qword ptr [rsp+20h],rbx 00000000`77352af5 57 push rdi 00000000`77352af6 4883ec30 sub rsp,30h 00000000`77352afa 448b4944 mov r9d,dword ptr [rcx+44h] 00000000`77352afe 0fb6fa movzx edi,dl 00000000`77352b01 488bd9 mov rbx,rcx 00000000`77352b04 4585c9 test r9d,r9d 00000000`77352b07 0f88a7000000 js ntdll!RtlAcquireResourceShared+0×65 (00000000`77352bb4) […]

We see the handle is taken from [RBX+28] and we see that RBX was saved at the function prologue and then the value of RCX was assigned to RBX. RCX as the first calling convention parameter should be a pointer to RTL_RESOURCE which has RTL_CRITICAL_SECTION as the first member and its size is 0×28:

0: kd> dt ntdll!_RTL_CRITICAL_SECTION ntdll!_RTL_CRITICAL_SECTION +0x000 DebugInfo : Ptr64 _RTL_CRITICAL_SECTION_DEBUG +0x008 LockCount : Int4B +0x00c RecursionCount : Int4B +0x010 OwningThread : Ptr64 Void +0x018 LockSemaphore : Ptr64 Void +0x020 SpinCount : Uint8B

Therefore [RBX+28] contains SharedSemaphore field which is assigned to RCX as a first parameter to NtWaitForSingleObject. The similar fragment of RtlAcquireResourceExclusive has [RBX+36] which 0×10 further than 0×28 and corresponds to ExclusiveSemaphore handle field:

ntdll!RtlAcquireResourceExclusive+0xd2: 00000000`770c2a12 488b4b38 mov rcx,qword ptr [rbx+38h] 00000000`770c2a16 4c8bc6 mov r8,rsi 00000000`770c2a19 33d2 xor edx,edx 00000000`770c2a1b e8d0e80400 call ntdll!NtWaitForSingleObject (00000000`771112f0) 00000000`770c2a20 3d02010000 cmp eax,102h 00000000`770c2a25 0f8401c60600 je ntdll! ?? ::FNODOBFM::`string’+0×12591 (00000000`7712f02c)

So we just need to know the vale of RBX and dump the structure to find OwningThread field. We can either calculate it from RSP or use /c switch with .frame command:

0: kd> kn *** Stack trace for last set context - .thread/.cxr resets it # Child-SP RetAddr Call Site 00 fffff880`0e870940 fffff800`01c76972 nt!KiSwapContext+0x7a 01 fffff880`0e870a80 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2 02 fffff880`0e870b10 fffff800`01f7b2be nt!KeWaitForSingleObject+0x19f 03 fffff880`0e870bb0 fffff800`01c801d3 nt!NtWaitForSingleObject+0xde 04 fffff880`0e870c20 00000000`773912fa nt!KiSystemServiceCopyEnd+0x13 05 00000000`022ae6c8 00000000`773470b4 ntdll!NtWaitForSingleObject+0xa 06 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0 07 00000000`022ae710 00000000`ff401675 lsm!CAutoSharedLock::CAutoSharedLock+0×61 08 00000000`022ae7e0 00000000`ff402c68 lsm!CTSSession::getTerminal+0×21 09 00000000`022ae820 000007fe`fd8bff85 lsm!RpcGetEnumResult+0×202 0a 00000000`022ae980 000007fe`fd8b4de2 RPCRT4!Invoke+0×65 0b 00000000`022ae9e0 000007fe`fd8b17bd RPCRT4!NdrStubCall2+0×32a 0c 00000000`022af000 000007fe`fd8b3254 RPCRT4!NdrServerCall2+0×1d 0d 00000000`022af030 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0×14 0e 00000000`022af060 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146 0f 00000000`022af180 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0×149 10 00000000`022af260 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0×20d 11 00000000`022af390 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf 12 00000000`022af4d0 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5 13 00000000`022af560 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0×26b 14 00000000`022af5f0 00000000`7713652d ntdll!TppWorkerThread+0×3f8 15 00000000`022af8f0 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd 16 00000000`022af920 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .frame /c 6 06 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0 rax=0000000000000000 rbx=00000000023ac128 rcx=0000000000000000 rdx=0000000000000000 rsi=0000000077472410 rdi=0000000000000001 rip=00000000773470b4 rsp=00000000022ae6d0 rbp=0000000000000000 r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=29406b2a1a85bd43 r13=0000000000000009 r14=000000000000000c r15=00000000022aef20 iopl=0 nv up di pl nz na pe nc cs=0000 ss=0000 ds=0000 es=0000 fs=0000 gs=0000 efl=00000000 ntdll!RtlAcquireResourceShared+0xd0: 00000000`773470b4 3d02010000 cmp eax,102h

0: kd> dp rbx+28 L10 00000000`023ac150 00000000`00001244 00000000`000001b5 00000000`023ac160 00000000`00000f3c ffffffff`00000000 00000000`023ac170 00000000`000021a0 00000000`00000000 00000000`023ac180 00000000`02735fc0 00000000`00000001 00000000`023ac190 00000000`00000000 01cf07ac`9fa06d27 00000000`023ac1a0 00000000`00000000 00000000`00000000 00000000`023ac1b0 ffffffff`ffffffff 00000000`00000000 00000000`023ac1c0 00000000`00000000 00000000`00000000

We check all these handles (OwnerThread seems comes earlier with NumberActive field missing but that could just differences between the old x86 structure implemented in ReactOS and x64 Windows):

0: kd> !handle 00000000`00001244

PROCESS fffffa804a16b260 SessionId: 0 Cid: 03c0 Peb: 7fffffdc000 ParentCid: 0350 DirBase: 195950000 ObjectTable: fffff8a0032424e0 HandleCount: 5252. Image: lsm.exe

Handle table at fffff8a0032424e0 with 5252 entries in use

1244: Object: fffffa804a79ad50 GrantedAccess: 00100003 Entry: fffff8a022b39910 Object: fffffa804a79ad50 Type: (fffffa8048fc8790) Semaphore ObjectHeader: fffffa804a79ad20 (new version) HandleCount: 1 PointerCount: 438

0: kd> !handle 00000000`00000f3c

PROCESS fffffa804a16b260 SessionId: 0 Cid: 03c0 Peb: 7fffffdc000 ParentCid: 0350 DirBase: 195950000 ObjectTable: fffff8a0032424e0 HandleCount: 5252. Image: lsm.exe

Handle table at fffff8a0032424e0 with 5252 entries in use

0f3c: Object: fffffa804fa81f60 GrantedAccess: 00100003 Entry: fffff8a02cd3ecf0 Object: fffffa804fa81f60 Type: (fffffa8048fc8790) Semaphore ObjectHeader: fffffa804fa81f30 (new version) HandleCount: 1 PointerCount: 1

0: kd> !thread -t 00000000`000021a0 3f THREAD fffffa804d5d51b0 Cid 03c0.21a0 Teb: 000007fffff9c000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable fffffa804d5d5578 Semaphore Limit 0×1 Waiting for reply to ALPC Message fffff8a02c9a9500 : queued at port fffffa804ac4e7d0 : owned by process fffffa804adc8730 Not impersonating DeviceMap fffff8a0000088c0 Owning Process fffffa804a16b260 Image: lsm.exe Attached Process N/A Image: N/A Wait Start TickCount 73337319 Ticks: 1466453 (0:06:21:53.328) Context Switch Count 69 IdealProcessor: 1 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address ntdll!TppWorkerThread (0×000000007735fbf0) Stack Init fffff8800aa1fdb0 Current fffff8800aa1f600 Base fffff8800aa20000 Limit fffff8800aa1a000 Call 0 Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Kernel stack not resident.

Child-SP RetAddr Call Site fffff880`0aa1f640 fffff800`01c76972 nt!KiSwapContext+0x7a fffff880`0aa1f780 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2 fffff880`0aa1f810 fffff800`01ca25af nt!KeWaitForSingleObject+0x19f fffff880`0aa1f8b0 fffff800`01f968b6 nt!AlpcpSignalAndWait+0x8f fffff880`0aa1f960 fffff800`01f95fb0 nt!AlpcpReceiveSynchronousReply+0x46 fffff880`0aa1f9c0 fffff800`01f93dab nt!AlpcpProcessSynchronousRequest+0x33d fffff880`0aa1fb00 fffff800`01c801d3 nt!NtAlpcSendWaitReceivePort+0x1ab fffff880`0aa1fbb0 00000000`77391b0a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0aa1fc20) 00000000`01dddb48 000007fe`fd8c8306 ntdll!ZwAlpcSendWaitReceivePort+0xa 00000000`01dddb50 000007fe`fd8c2a02 RPCRT4!LRPC_CCALL::SendReceive+0x156 00000000`01dddc10 000007fe`ff5b28c0 RPCRT4!I_RpcSendReceive+0x42 00000000`01dddc40 000007fe`ff5b282f ole32!ThreadSendReceive+0x40 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 5003] 00000000`01dddc90 000007fe`ff5b265b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 4454] 00000000`01dddd30 000007fe`ff46daaa ole32!CRpcChannelBuffer::SendReceive2+0x11b [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 4074] 00000000`01dddef0 000007fe`ff46da0c ole32!CAptRpcChnl::SendReceive+0x52 [d:\w7rtm\com\ole32\com\dcomrem\callctrl.cxx @ 603] 00000000`01dddfc0 000007fe`ff5b205d ole32!CCtxComChnl::SendReceive+0x68 [d:\w7rtm\com\ole32\com\dcomrem\ctxchnl.cxx @ 734] 00000000`01dde070 000007fe`fd96b949 ole32!NdrExtpProxySendReceive+0x45 [d:\w7rtm\com\rpc\ndrole\proxy.cxx @ 1932] 00000000`01dde0a0 000007fe`ff5b21d0 RPCRT4!NdrpClientCall3+0x2e2 00000000`01dde360 000007fe`ff46d8a2 ole32!ObjectStublessClient+0x11d [d:\w7rtm\com\rpc\ndrole\amd64\stblsclt.cxx @ 621] 00000000`01dde6f0 00000000`ff417d26 ole32!ObjectStubless+0x42 [d:\w7rtm\com\rpc\ndrole\amd64\stubless.asm @ 117] 00000000`01dde740 00000000`ff4186ba lsm!CTSSession::Disconnect+0x3a5 00000000`01dde810 000007fe`fd8bff85 lsm!RpcDisconnect+0x15e 00000000`01dde850 000007fe`fd96b68e RPCRT4!Invoke+0x65 00000000`01dde8a0 000007fe`fd8a92e0 RPCRT4!Ndr64StubWorker+0x61b 00000000`01ddee60 000007fe`fd8b3254 RPCRT4!NdrServerCallAll+0x40 00000000`01ddeeb0 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0x14 00000000`01ddeee0 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x146 00000000`01ddf000 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0x149 00000000`01ddf0e0 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0x20d 00000000`01ddf210 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0x3bf 00000000`01ddf350 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5 00000000`01ddf3e0 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0x26b 00000000`01ddf470 00000000`7713652d ntdll!TppWorkerThread+0x3f8 00000000`01ddf770 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd 00000000`01ddf7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

We see the wait chain continues with waiting for an ALPC request.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Assembly Language, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 209)

Saturday, September 6th, 2014

The availability of direct dump modification raises the possibility of Tampered Dumps. These are memory dumps specifically modified to alter structural and behavioural diagnostic patterns, for example, to suppress certain module involvement or introduce fictitious past objects and interaction traces such as Execution Residue and Module Hints. There can be 2 types of such artefacts: strong tampering with new or altered information completely integrated into memory fabric and weak tampering to confuse inexperienced software support engineers and memory forensics analysts.

For example, in one such experimental process memory dump we see Exception Stack Trace pointing to a problem in calc module:

0:003> k Child-SP RetAddr Call Site 00000000`0244e858 000007fe`fd061430 ntdll!NtWaitForMultipleObjects+0xa 00000000`0244e860 00000000`76ec1723 KERNELBASE!WaitForMultipleObjectsEx+0xe8 00000000`0244e960 00000000`76f3b5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3 00000000`0244e9f0 00000000`76f3b767 kernel32!WerpReportFaultInternal+0x215 00000000`0244ea90 00000000`76f3b7bf kernel32!WerpReportFault+0x77 00000000`0244eac0 00000000`76f3b9dc kernel32!BasepReportFault+0x1f 00000000`0244eaf0 00000000`77153398 kernel32!UnhandledExceptionFilter+0x1fc 00000000`0244ebd0 00000000`770d85c8 ntdll! ?? ::FNODOBFM::`string'+0x2365 00000000`0244ec00 00000000`770e9d2d ntdll!_C_specific_handler+0x8c 00000000`0244ec70 00000000`770d91cf ntdll!RtlpExecuteHandlerForException+0xd 00000000`0244eca0 00000000`77111248 ntdll!RtlDispatchException+0x45a 00000000`0244f380 00000000`ffdbdb27 ntdll!KiUserExceptionDispatch+0×2e 00000000`0244fab0 00000000`76eb59ed calc!CTimedCalc::WatchDogThread+0xb2 00000000`0244faf0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd 00000000`0244fb20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The default analysis command (!analyse -v) diagnoses “stack corruption”:

FAULTING_IP: kernel32!UnhandledExceptionFilter+1fc 00000000`76f3b9dc 448bf0 mov r14d,eax

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 0000000076f3b9dc (kernel32!UnhandledExceptionFilter+0x00000000000001fc) ExceptionCode: 0244e9f0 ExceptionFlags: 00000000 NumberParameters: 0

DEFAULT_BUCKET_ID: STACK_CORRUPTION

PRIMARY_PROBLEM_CLASS: STACK_CORRUPTION

BUGCHECK_STR: APPLICATION_FAULT_STACK_CORRUPTION

IP_ON_HEAP: 8d483674c33bfffa The fault address in not in any loaded module, please check your build's rebase log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may contain the address if it were loaded.

UNALIGNED_STACK_POINTER: 0000000076f3b767

STACK_TEXT: 00000000`00000000 00000000`00000000 calc!CTimedCalc::WatchDogThread+0x0

FOLLOWUP_IP: calc!CTimedCalc::WatchDogThread+0 00000000`ffd92254 48895c2408 mov qword ptr [rsp+8],rbx

Stored Exception resembles signs of Local Buffer Overflow (segment register values and CPU flags have suspiciously invalid values, possibly Lateral Damage):

0:003> .ecxr rax=0000000000000000 rbx=0000000000000001 rcx=000000000244ec30 rdx=000000000244ec30 rsi=0100000000000080 rdi=0000000000000158 rip=0000000076f3b9dc rsp=0000000076f3b767 rbp=0000000000000000 r8=0000000000000000 r9=ffffffffffffffff r10=0000000076f3b7bf r11=000000000244ec30 r12=0000000000000001 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up di pl nz na pe nc cs=0000 ss=0000 ds=0266 es=0000 fs=0000 gs=0154 efl=00000000 kernel32!UnhandledExceptionFilter+0×1fc: 00000000`76f3b9dc 448bf0 mov r14d,eax

0:003> k *** Stack trace for last set context - .thread/.cxr resets it Child-SP RetAddr Call Site 00000000`76f3b767 8d483674`c33bfffa kernel32!UnhandledExceptionFilter+0x1fc 00000000`76f3b847 5aa3e800`05bfac0d 0x8d483674`c33bfffa 00000000`76f3b84f ebffcf83`48ccfff9 0x5aa3e800`05bfac0d 00000000`76f3b857 8348c000`0409ba27 0xebffcf83`48ccfff9 00000000`76f3b85f 54dfe8cf`8b48ffcf 0x8348c000`0409ba27 00000000`76f3b867 4c02778d`db33fff9 0x54dfe8cf`8b48ffcf 00000000`76f3b86f 4c000000`e024a48b 0x4c02778d`db33fff9 00000000`76f3b877 ffcf8348`04ebeb8b 0x4c000000`e024a48b 00000000`76f3b87f fffc59e9`e8cc8b49 0xffcf8348`04ebeb8b 00000000`76f3b887 42e9c78b`0775c73b 0xfffc59e9`e8cc8b49 00000000`76f3b88f fffa6fa9`e8000003 0x42e9c78b`0775c73b 00000000`76f3b897 32e9c033`0774c33b 0xfffa6fa9`e8000003 00000000`76f3b89f fa7f3d8d`4c000003 0x32e9c033`0774c33b 00000000`76f3b8a7 de15ffcf`8b490006 0xfa7f3d8d`4c000003 00000000`76f3b8af f9370d8b`4800000e 0xde15ffcf`8b490006 00000000`76f3b8b7 000014a1`15ff0006 0xf9370d8b`4800000e 00000000`76f3b8bf 840fc33b`48f08b4c 0x000014a1`15ff0006 00000000`76f3b8c7 f6158b48`00000099 0x840fc33b`48f08b4c 00000000`76f3b8cf 0238c281`480006f3 0xf6158b48`00000099 00000000`76f3b8d7 48cfe8c8`8b480000 0x0238c281`480006f3 00000000`76f3b8df 8b4c7f74`c33bfff9 0x48cfe8c8`8b480000 00000000`76f3b8e7 888b4900`06f3dc05 0x8b4c7f74`c33bfff9 00000000`76f3b8ef 75083949`00000238 0x888b4900`06f3dc05 00000000`76f3b8f7 00000240`808b496c 0x75083949`00000238 00000000`76f3b8ff 8b415f75`08403949 0x00000240`808b496c 00000000`76f3b907 00024880`3b411040 0x8b415f75`08403949 00000000`76f3b90f 01040000`a9527500 0x00024880`3b411040 00000000`76f3b917 00025090`8d491874 0x01040000`a9527500 00000000`76f3b91f c68a4418`488d4900 0x00025090`8d491874 00000000`76f3b927 c33a0000`117315ff 0xc68a4418`488d4900 00000000`76f3b92f 4e15ffcf`8b493374 0xc33a0000`117315ff 00000000`76f3b937 ff41cc8b`4900000e 0x4e15ffcf`8b493374 00000000`76f3b93f 00028c84`0fc63bd6 0xff41cc8b`4900000e 00000000`76f3b947 00028484`0fc73b00 0x00028c84`0fc63bd6 00000000`76f3b94f 6ee7e819`75c33b00 0x00028484`0fc73b00 00000000`76f3b957 c0331074`c33bfffa 0x6ee7e819`75c33b00 00000000`76f3b95f cf8b4900`000270e9 0xc0331074`c33bfffa 00000000`76f3b967 8b490000`0e1b15ff 0xcf8b4900`000270e9 00000000`76f3b96f 3b000013`e215ffcc 0x8b490000`0e1b15ff 00000000`76f3b977 0253e9c7`8b0775c7 0x3b000013`e215ffcc 00000000`76f3b97f 41fff959`4ae80000 0x0253e9c7`8b0775c7 00000000`76f3b987 c6844100`000002be 0x41fff959`4ae80000 00000000`76f3b98f 15ff0000`023d850f 0xc6844100`000002be 00000000`76f3b997 850f20a8`00000f65 0x15ff0000`023d850f 00000000`76f3b99f 245c8948`0000022f 0x850f20a8`00000f65 00000000`76f3b9a7 448d4c3e`4e8d4520 0x245c8948`0000022f 00000000`76f3b9af ffc933d6`8b416024 0x448d4c3e`4e8d4520 00000000`76f3b9b7 7cc33b00`0009f415 0xffc933d6`8b416024 00000000`76f3b9bf 730a7024`64ba0f0f 0x7cc33b00`0009f415 00000000`76f3b9c7 00000205`e9c68b07 0x730a7024`64ba0f0f 00000000`76f3b9cf cc8b49d6`8bfb8b44 0x00000205`e9c68b07 00000000`76f3b9d7 f08b44ff`fffdc4e8 0xcc8b49d6`8bfb8b44 00000000`76f3b9df e9c03307`7508f883 0xf08b44ff`fffdc4e8 00000000`76f3b9e7 7506f883`000001e9 0xe9c03307`7508f883 00000000`76f3b9ef c33bfffa`6e4be810 0x7506f883`000001e9 00000000`76f3b9f7 0001d4e9`c0330774 0xc33bfffa`6e4be810 00000000`76f3b9ff 86850f04`fe834100 0x0001d4e9`c0330774 00000000`76f3ba07 0000024a`ba000001 0x86850f04`fe834100 00000000`76f3ba0f 00b841ce`8b45c933 0x0000024a`ba000001 00000000`76f3ba17 fff7a249`e8000010 0x00b841ce`8b45c933 00000000`76f3ba1f 0775c33b`48e88b4c 0xfff7a249`e8000010 00000000`76f3ba27 48000001`a6e9c033 0x0775c33b`48e88b4c 00000000`76f3ba2f 24448948`3024448d 0x48000001`a6e9c033 00000000`76f3ba37 0000f024`8c8d4c20 0x24448948`3024448d 00000000`76f3ba3f 49000001`25b84100 0x0000f024`8c8d4c20 00000000`76f3ba47 8a0fe8cf`8b48d58b 0x49000001`25b84100 00000000`76f3ba4f 4166097c`c33bfffe 0x8a0fe8cf`8b48d58b 00000000`76f3ba57 39fe450f`44005d39 0x4166097c`c33bfffe 00000000`76f3ba5f 850f0000`00f0249c 0x39fe450f`44005d39 00000000`76f3ba67 240c8b49`000000bc 0x850f0000`00f0249c 00000000`76f3ba6f 40244489`48016348 0x240c8b49`000000bc 00000000`76f3ba77 24448948`10418b48 0x40244489`48016348 00000000`76f3ba7f 75c00000`06398148 0x24448948`10418b48 00000000`76f3ba87 480b7203`18798318 0x75c00000`06398148 00000000`76f3ba8f 50244489`4830418b 0x480b7203`18798318 00000000`76f3ba97 eb50245c`89481ceb 0x50244489`4830418b 00000000`76f3ba9f 8b480b72`18713915 0xeb50245c`89481ceb 00000000`76f3baa7 eb502444`89482041 0x8b480b72`18713915 00000000`76f3baaf 02ba5024`5c894805 0xeb502444`89482041 00000000`76f3bab7 0b721851`39000000 0x02ba5024`5c894805 00000000`76f3babf 24448948`28418b48 0x0b721851`39000000 00000000`76f3bac7 58245c89`4805eb58 0x24448948`28418b48 00000000`76f3bacf ba1d3808`74fb3b44 0x58245c89`4805eb58 00000000`76f3bad7 48d68b02`740006fd 0xba1d3808`74fb3b44 00000000`76f3badf 48000000`e824848d 0x48d68b02`740006fd 00000000`76f3bae7 20245489`28244489 0x48000000`e824848d 00000000`76f3baef c0334540`244c8d4c 0x20245489`28244489 00000000`76f3baf7 000144b9`04508d41 0xc0334540`244c8d4c 00000000`76f3baff ba00000d`7215ffd0 0x000144b9`04508d41 00000000`76f3bb07 8c8bc223`c0000000 0xba00000d`7215ffd0 00000000`76f3bb0f b8c23b00`0000e824 0x8c8bc223`c0000000 00000000`76f3bb17 89c8440f`00000006 0xb8c23b00`0000e824 00000000`76f3bb1f 07eb0000`00e8248c 0x89c8440f`00000006 00000000`76f3bb27 44000000`e8248c8b 0x07eb0000`00e8248c 00000000`76f3bb2f 7403f983`5d74fb3b 0x44000000`e8248c8b 00000000`76f3bb37 000000f0`249c3909 0x7403f983`5d74fb3b 00000000`76f3bb3f 0006fd4d`058a4f74 0x000000f0`249c3909 00000000`76f3bb47 f85f5ce8`4b75c33a 0x0006fd4d`058a4f74 00000000`76f3bb4f 448b3b75`5c5838ff 0xf85f5ce8`4b75c33a 00000000`76f3bb57 894c2824`44893024 0x448b3b75`5c5838ff 00000000`76f3bb5f 08244c8b`4d20246c 0x894c2824`44893024 00000000`76f3bb67 fec2c748`24048b4d 0x08244c8b`4d20246c 00000000`76f3bb6f b6e8cf8b`48ffffff 0xfec2c748`24048b4d 00000000`76f3bb77 fd130db6`0fffffea 0xb6e8cf8b`48ffffff 00000000`76f3bb7f 88ce4c0f`c33b0006 0xfd130db6`0fffffea 00000000`76f3bb87 ebfb8b00`06fd080d 0x88ce4c0f`c33b0006 00000000`76f3bb8f 3a0006fc`fe058a29 0xebfb8b00`06fd080d 00000000`76f3bb97 8b240c8b`491874c3 0x3a0006fc`fe058a29 00000000`76f3bb9f 060f15ff`cf8b4811 0x8b240c8b`491874c3 00000000`76f3bba7 0000f824`bc8b0000 0x060f15ff`cf8b4811 00000000`76f3bbaf 00f824bc`8b07eb00 0x0000f824`bc8b0000 00000000`76f3bbb7 331074eb`3b4c0000 0x00f824bc`8b07eb00 00000000`76f3bbbf 49000080`00b841d2 0x331074eb`3b4c0000 00000000`76f3bbc7 8bfff74b`5ae8cd8b 0x49000080`00b841d2 00000000`76f3bbcf c48148c6`8b02ebc7 0x8bfff74b`5ae8cd8b 00000000`76f3bbd7 5e415f41`000000a0 0xc48148c6`8b02ebc7 00000000`76f3bbdf c35b5e5f`5c415d41 0x5e415f41`000000a0 00000000`76f3bbe7 158ead00`00000090 0xc35b5e5f`5c415d41 00000000`76f3bbef 00000200`00000053 0x158ead00`00000090 00000000`76f3bbf7 09bc2400`00002500 0x00000200`00000053 00000000`76f3bbff 00000000`09b42400 0x09bc2400`00002500 00000000`76f3bc07 7e023553`158ead00 0x9b42400 00000000`76f3bc0f 00000400`00000a19 0x7e023553`158ead00 00000000`76f3bc17 09b42000`09bc2000 0x00000400`00000a19 00000000`76f3bc1f 445352bb`03197e00 0x09b42000`09bc2000 00000000`76f3bc27 4c886225`48e28953 0x445352bb`03197e00 00000000`76f3bc2f 4fb29af4`dfbb8344 0x4c886225`48e28953 00000000`76f3bc37 72656b00`0000020e 0x4fb29af4`dfbb8344 00000000`76f3bc3f 64702e32`336c656e 0x72656b00`0000020e 00000000`76f3bc47 00000000`00000062 0x64702e32`336c656e

We check for any Hidden Exceptions and find it was NULL Data Pointer:

0:003> .cxr Resetting default scope

0:003> k Child-SP RetAddr Call Site 00000000`0244e858 000007fe`fd061430 ntdll!NtWaitForMultipleObjects+0xa 00000000`0244e860 00000000`76ec1723 KERNELBASE!WaitForMultipleObjectsEx+0xe8 00000000`0244e960 00000000`76f3b5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3 00000000`0244e9f0 00000000`76f3b767 kernel32!WerpReportFaultInternal+0x215 00000000`0244ea90 00000000`76f3b7bf kernel32!WerpReportFault+0x77 00000000`0244eac0 00000000`76f3b9dc kernel32!BasepReportFault+0x1f 00000000`0244eaf0 00000000`77153398 kernel32!UnhandledExceptionFilter+0x1fc 00000000`0244ebd0 00000000`770d85c8 ntdll! ?? ::FNODOBFM::`string'+0x2365 00000000`0244ec00 00000000`770e9d2d ntdll!_C_specific_handler+0x8c 00000000`0244ec70 00000000`770d91cf ntdll!RtlpExecuteHandlerForException+0xd 00000000`0244eca0 00000000`77111248 ntdll!RtlDispatchException+0×45a 00000000`0244f380 00000000`ffdbdb27 ntdll!KiUserExceptionDispatch+0×2e 00000000`0244fab0 00000000`76eb59ed calc!CTimedCalc::WatchDogThread+0xb2 00000000`0244faf0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd 00000000`0244fb20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:003> dps 00000000`0244eca0 00000000`0244fab0 00000000`0244eca0 00000000`02450000 00000000`0244eca8 00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0) 00000000`0244ecb0 00000000`00012f00 00000000`0244ecb8 00000000`7711920a ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3da 00000000`0244ecc0 00000000`00000005 00000000`0244ecc8 00000000`00000000 00000000`0244ecd0 00000000`00000000 00000000`0244ecd8 00000000`00000000 00000000`0244ece0 00000000`0244fb20 00000000`0244ece8 00000000`00000000 00000000`0244ecf0 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`0244ecf8 00000000`00000000 00000000`0244ed00 00000000`00000000 00000000`0244ed08 00000000`02450000 00000000`0244ed10 00000000`771e8180 ntdll!`string'+0xc040 00000000`0244ed18 00000000`0244b000 00000000`0244ed20 00000000`0244f250 00000000`0244ed28 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`0244ed30 00000000`770ec541 ntdll!RtlUserThreadStart+0x1d 00000000`0244ed38 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`0244ed40 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`0244ed48 00000000`0244fb20 00000000`0244ed50 00000000`771d7718 ntdll!LdrpDefaultExtension 00000000`0244ed58 00000000`0244ed80 00000000`0244ed60 00000000`770d852c ntdll!_C_specific_handler 00000000`0244ed68 00000000`771e8180 ntdll!`string'+0xc040 00000000`0244ed70 00000000`0244f250 00000000`0244ed78 00000000`00000000 00000000`0244ed80 00000000`00000000 00000000`0244ed88 00000000`00000000 00000000`0244ed90 00000000`00000000 00000000`0244ed98 00000000`00000000 00000000`0244eda0 00000000`00000000 00000000`0244eda8 00000000`00000000 00000000`0244edb0 00001f80`00000000 00000000`0244edb8 00000000`00000033 00000000`0244edc0 00010246`002b0000 00000000`0244edc8 00000000`00000000 00000000`0244edd0 00000000`00000000 00000000`0244edd8 00000000`00000000 00000000`0244ede0 00000000`00000000 00000000`0244ede8 000007fe`ff3625c0 msctf!s_szCompClassName 00000000`0244edf0 00000000`00200000 00000000`0244edf8 00000000`0244ee40 00000000`0244ee00 00000000`0244ee40 00000000`0244ee08 00000000`0244ee40 00000000`0244ee10 00000000`00000000 00000000`0244ee18 00000000`0244fb70 00000000`0244ee20 00000000`00000000 00000000`0244ee28 00000000`00000000 00000000`0244ee30 00000000`00000000 00000000`0244ee38 000007fe`fd602790 ole32!`string' 00000000`0244ee40 00000000`00292170 00000000`0244ee48 00000000`770e7a33 ntdll!LdrpFindOrMapDll+0x138 00000000`0244ee50 00000000`0244ef68 00000000`0244ee58 00000000`00000000 00000000`0244ee60 00000000`00000000 00000000`0244ee68 00000000`00000000 00000000`0244ee70 00000000`00000000 00000000`0244ee78 00000000`00000000 00000000`0244ee80 00000000`0000027f 00000000`0244ee88 00000000`00000000 00000000`0244ee90 00000000`00000000 00000000`0244ee98 0000ffff`00001f80 00000000`0244eea0 00000000`00000000 00000000`0244eea8 00000000`00000000 00000000`0244eeb0 00000000`00000000 00000000`0244eeb8 00000000`00000000 00000000`0244eec0 00000000`00000000 00000000`0244eec8 00000000`00000000 00000000`0244eed0 00000000`00000000 00000000`0244eed8 00000000`00000000 00000000`0244eee0 00000000`00000000 00000000`0244eee8 00000000`00000000 00000000`0244eef0 00000000`00000000 00000000`0244eef8 00000000`00000000 00000000`0244ef00 00000000`00000000 00000000`0244ef08 00000000`00000000 00000000`0244ef10 00000000`00000000 00000000`0244ef18 00000000`00000000 00000000`0244ef20 00000000`00000000 00000000`0244ef28 00000000`771192a8 ntdll!LdrpApplyFileNameRedirection+0x2d3 00000000`0244ef30 00000000`00000000 00000000`0244ef38 00000000`00000000 00000000`0244ef40 00000000`00000000 00000000`0244ef48 00000000`02080000 00000000`0244ef50 00000000`0244f028 00000000`0244ef58 00000000`0244f020 00000000`0244ef60 00000000`00000000 00000000`0244ef68 00000000`00000000 00000000`0244ef70 00000000`00000000 00000000`0244ef78 000007fe`fd602848 ole32!`string' 00000000`0244ef80 00000000`00000000 00000000`0244ef88 00000000`00000000 00000000`0244ef90 00000000`00000000 00000000`0244ef98 00000000`00000000 00000000`0244efa0 00000000`00000000 00000000`0244efa8 00000000`00000000 00000000`0244efb0 00000000`00000000 00000000`0244efb8 00000000`00000000 00000000`0244efc0 00000000`00000000 00000000`0244efc8 00000000`00000000 00000000`0244efd0 00000000`00000000 00000000`0244efd8 00000000`00000000 00000000`0244efe0 00000000`00000000 00000000`0244efe8 00000000`00000000 00000000`0244eff0 00000000`00000000 00000000`0244eff8 00000000`00000000 00000000`0244f000 00000000`00000000 00000000`0244f008 00000000`00000000 00000000`0244f010 00000000`00000000 00000000`0244f018 00000000`00000000 00000000`0244f020 00000000`0244f038 00000000`0244f028 00000000`0000011b 00000000`0244f030 00000000`024d0000 00000000`0244f038 00000080`001a024d 00000000`0244f040 00000000`01c0c8a0 00000000`0244f048 00000000`002f0101 00000000`0244f050 00000000`00000000 00000000`0244f058 00000000`00000022 00000000`0244f060 00000000`002f9b00 00000000`0244f068 00000000`01bd5390 00000000`0244f070 00000000`002f7c00 00000000`0244f078 00000000`01bd5580 00000000`0244f080 00000000`01bd57b0 00000000`0244f088 00000000`002f9b00 00000000`0244f090 00000000`00000000 00000000`0244f098 00000024`00000003 00000000`0244f0a0 00000000`002e91b0 00000000`0244f0a8 00000000`00000022 00000000`0244f0b0 00000000`771d5430 ntdll!RtlpInterceptorRoutines 00000000`0244f0b8 00000000`00000000 00000000`0244f0c0 00000000`00000010 00000000`0244f0c8 00000000`01bd0000 00000000`0244f0d0 00000000`00000008 00000000`0244f0d8 00000000`00000001 00000000`0244f0e0 00000000`01bd0288 00000000`0244f0e8 00000000`77113448 ntdll!RtlAllocateHeap+0xe4 00000000`0244f0f0 00000000`00000000 00000000`0244f0f8 00000000`00000001 00000000`0244f100 000002b2`000f002f 00000000`0244f108 00000000`01bd5780 00000000`0244f110 00000000`00250230 00000000`0244f118 00000000`000000df 00000000`0244f120 00000000`002551a0 00000000`0244f128 00000000`00255210 00000000`0244f130 00000000`002f9b00 00000000`0244f138 00000000`002551a0 00000000`0244f140 00000000`000000df 00000000`0244f148 00000000`10000010 00000000`0244f150 00000000`00250230 00000000`0244f158 00000000`00000000 00000000`0244f160 00000000`00250498 00000000`0244f168 00000000`0025026c 00000000`0244f170 00000000`002f9b00 00000000`0244f178 00000000`002551a0 00000000`0244f180 00000000`00000022 00000000`0244f188 00000000`76fd88b8 user32!GetPropW+0x4d 00000000`0244f190 00000000`00002974 00000000`0244f198 00000000`76fd88b8 user32!GetPropW+0x4d 00000000`0244f1a0 00000000`00250230 00000000`0244f1a8 00000000`76fd7931 user32!IsWindow+0x9 00000000`0244f1b0 00000000`002ed6d0 00000000`0244f1b8 00000000`76fd7931 user32!IsWindow+0x9 00000000`0244f1c0 00000000`00000000 00000000`0244f1c8 00000000`01c0c8d0 00000000`0244f1d0 00000000`01c0c8a0 00000000`0244f1d8 00000000`00000000 00000000`0244f1e0 00000000`00000008 00000000`0244f1e8 00000000`01bd0000 00000000`0244f1f0 00000000`00000000 00000000`0244f1f8 00000000`770f41c8 ntdll!RtlpReAllocateHeap+0x178 00000000`0244f200 00000000`00000002 00000000`0244f208 00000000`00000002 00000000`0244f210 00000000`00000000 00000000`0244f218 000007fe`4f00024d 00000000`0244f220 00000000`00000000 00000000`0244f228 000007fe`fb601381 uxtheme!CThemeWnd::_PreDefWindowProc+0x31 00000000`0244f230 00000000`00000082 00000000`0244f238 00000000`00000000 00000000`0244f240 00000000`7a337100 00000000`0244f248 00000000`01c0c8c0 00000000`0244f250 00000000`00000003 00000000`0244f258 00000000`76eb59e0 kernel32!BaseThreadInitThunk 00000000`0244f260 00000000`ffdbdb32 calc!CTimedCalc::Start+0xa9 00000000`0244f268 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0) 00000000`0244f270 00000000`ffe0ac64 calc!_dyn_tls_init_callback <PERF> (calc+0x7ac64) 00000000`0244f278 00000000`76ea0000 kernel32!TestResourceDataMatchEntry <PERF> (kernel32+0x0) 00000000`0244f280 00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0) 00000000`0244f288 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0) 00000000`0244f290 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0) 00000000`0244f298 00000000`76fd760e user32!RealDefWindowProcW+0x5a 00000000`0244f2a0 00000000`00000001 00000000`0244f2a8 000007fe`fb600037 uxtheme!operator delete <PERF> (uxtheme+0x37) 00000000`0244f2b0 00000000`01bd0158 00000000`0244f2b8 00000000`00000082 00000000`0244f2c0 00000000`00000000 00000000`0244f2c8 00000000`00000003 00000000`0244f2d0 00000000`000111f2 00000000`0244f2d8 00000000`00000054 00000000`0244f2e0 00000000`00000000 00000000`0244f2e8 00000000`00000000 00000000`0244f2f0 00000000`00000001 00000000`0244f2f8 00000000`01c11c60 00000000`0244f300 00000000`0244f462 00000000`0244f308 00000000`01bd0230 00000000`0244f310 00000000`00000000 00000000`0244f318 00000000`00000000 00000000`0244f320 00000000`00000000 00000000`0244f328 00000000`14010015 00000000`0244f330 00000000`01c11570 00000000`0244f338 00000000`00000000 00000000`0244f340 00000000`00000000 00000000`0244f348 00000000`00000000 00000000`0244f350 00000000`00009c40 00000000`0244f358 00000000`00000000 00000000`0244f360 00000000`00000000 00000000`0244f368 00000000`00000000 00000000`0244f370 00000000`00002710 00000000`0244f378 00000000`77111248 ntdll!KiUserExceptionDispatch+0×2e 00000000`0244f380 00000000`0244f870 00000000`0244f388 00000000`0244f380 00000000`0244f390 00000000`00000000 00000000`0244f398 00000000`00000000 00000000`0244f3a0 000007fe`fb63fb40 uxtheme!$$VProc_ImageExportDirectory 00000000`0244f3a8 00000000`00000ad5 00000000`0244f3b0 00001f80`0010005f 00000000`0244f3b8 0053002b`002b0033 00000000`0244f3c0 00010246`002b002b 00000000`0244f3c8 00000000`00000000 00000000`0244f3d0 00000000`00000000 00000000`0244f3d8 00000000`00000000 00000000`0244f3e0 00000000`00000000 00000000`0244f3e8 00000000`00000000 00000000`0244f3f0 00000000`00000000 00000000`0244f3f8 00000000`0012c770 00000000`0244f400 00000000`00000000 00000000`0244f408 00000000`00000000 00000000`0244f410 00000000`00002710 00000000`0244f418 00000000`0244fab0 00000000`0244f420 00000000`00000000 00000000`0244f428 00000000`00000000 00000000`0244f430 00000000`00000000 00000000`0244f438 00000000`0244f938 00000000`0244f440 00000000`00962210 00000000`0244f448 00000000`00000000 00000000`0244f450 00000000`0244f9a0 00000000`0244f458 00000000`00009c40 00000000`0244f460 00000000`00000000 00000000`0244f468 00000000`00000000 00000000`0244f470 00000000`00000000 00000000`0244f478 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2 00000000`0244f480 00000000`0000027f 00000000`0244f488 00000000`00000000 00000000`0244f490 00000000`00000000 00000000`0244f498 0000ffff`00001f80 00000000`0244f4a0 00000000`00000000 00000000`0244f4a8 00000000`00000000 00000000`0244f4b0 00000000`00000000 00000000`0244f4b8 00000000`00000000 00000000`0244f4c0 00000000`00000000 00000000`0244f4c8 00000000`00000000 00000000`0244f4d0 00000000`00000000 00000000`0244f4d8 00000000`00000000 00000000`0244f4e0 00000000`00000000 00000000`0244f4e8 00000000`00000000 00000000`0244f4f0 00000000`00000000 00000000`0244f4f8 00000000`00000000 00000000`0244f500 00000000`00000000 00000000`0244f508 00000000`00000000 00000000`0244f510 00000000`00000000 00000000`0244f518 00000000`00000000 00000000`0244f520 00000000`00000000 00000000`0244f528 00000000`00000000 00000000`0244f530 00000000`00000000 00000000`0244f538 00000000`00000000 00000000`0244f540 00000000`00000000 00000000`0244f548 00000000`00000000 00000000`0244f550 00000000`00000000 00000000`0244f558 00000000`00000000 00000000`0244f560 00000000`00000000 00000000`0244f568 00000000`00000000 00000000`0244f570 00000000`00000000 00000000`0244f578 00000000`00000000 00000000`0244f580 00000000`00000000 00000000`0244f588 00000000`00000000 00000000`0244f590 00000000`00000000 00000000`0244f598 00000000`00000000 00000000`0244f5a0 00000000`00000000 00000000`0244f5a8 00000000`00000000 00000000`0244f5b0 00000000`00000000 00000000`0244f5b8 00000000`00000000 00000000`0244f5c0 00000000`00000000 00000000`0244f5c8 00000000`00000000 00000000`0244f5d0 00000000`00000000 00000000`0244f5d8 00000000`00000000 00000000`0244f5e0 00000000`00000000 00000000`0244f5e8 00000000`00000000 00000000`0244f5f0 00000000`00000000 00000000`0244f5f8 00000000`00000000 00000000`0244f600 00000000`00000000 00000000`0244f608 00000000`00000000 00000000`0244f610 00000000`00000000 00000000`0244f618 00000000`00000000 00000000`0244f620 00000000`00000000 00000000`0244f628 00000000`00000000 00000000`0244f630 00000000`00000000 00000000`0244f638 00000000`00000000 00000000`0244f640 00000000`00000000 00000000`0244f648 00000000`00000000 00000000`0244f650 00000000`00000000 00000000`0244f658 00000000`00000000 00000000`0244f660 00000000`00000000 00000000`0244f668 fffff800`032d5e53 00000000`0244f670 00000000`00000002 00000000`0244f678 00000000`00000000 00000000`0244f680 00000000`01c11580 00000000`0244f688 00000000`00000082 00000000`0244f690 00000000`00000082 00000000`0244f698 00000000`000111e4 00000000`0244f6a0 00000000`00000002 00000000`0244f6a8 00000000`0244f6f0 00000000`0244f6b0 00000000`00000002 00000000`0244f6b8 00000000`00000000 00000000`0244f6c0 00000000`000111e4 00000000`0244f6c8 00000000`00000000 00000000`0244f6d0 00000000`00000082 00000000`0244f6d8 00000000`00000000 00000000`0244f6e0 00000000`00000000 00000000`0244f6e8 00000000`76fe76c2 user32!DefDlgProcW+0×36 00000000`0244f6f0 00000000`00000000 00000000`0244f6f8 00000000`00000000 00000000`0244f700 00000000`000111e4 00000000`0244f708 00000000`00000000 00000000`0244f710 00000000`00000082 00000000`0244f718 00000000`00000000 00000000`0244f720 00000000`0244f908 00000000`0244f728 00000000`76fd9bef user32!UserCallWinProcCheckWow+0×1cb 00000000`0244f730 00000000`00962210 00000000`0244f738 00000000`00000001 00000000`0244f740 00000000`00000000 00000000`0244f748 00000000`00000000 00000000`0244f750 00000000`0244f768 00000000`0244f758 00000000`0244f778 00000000`0244f760 00000000`00000001 00000000`0244f768 00000000`00000000 00000000`0244f770 00000000`00000000 00000000`0244f778 00000000`00000000 00000000`0244f780 00000000`00000048 00000000`0244f788 00000000`00000001 00000000`0244f790 00000000`00000000 00000000`0244f798 00000000`00000000 00000000`0244f7a0 00000000`00000070 00000000`0244f7a8 ffffffff`ffffffff 00000000`0244f7b0 ffffffff`ffffffff 00000000`0244f7b8 00000000`76fd9b43 user32!UserCallWinProcCheckWow+0×99 00000000`0244f7c0 00000000`76fd9bef user32!UserCallWinProcCheckWow+0×1cb 00000000`0244f7c8 00000000`00000000 00000000`0244f7d0 00000000`00000000 00000000`0244f7d8 00000000`00000000 00000000`0244f7e0 00000000`00000000 00000000`0244f7e8 00000000`76fd72cb user32!DispatchClientMessage+0xc3 00000000`0244f7f0 00000000`00000000 00000000`0244f7f8 00000000`770e46b4 ntdll!NtdllDialogWndProc_W 00000000`0244f800 00000000`00000000 00000000`0244f808 00000000`00000000 00000000`0244f810 00000000`00000000 00000000`0244f818 00000000`00000000 00000000`0244f820 00000000`00962238 00000000`0244f828 00000000`00000001 00000000`0244f830 00000000`00000000 00000000`0244f838 00000000`00000000 00000000`0244f840 00000000`00000000 00000000`0244f848 00000000`00000000 00000000`0244f850 00000730`fffffb30 00000000`0244f858 000004d0`fffffb30 00000000`0244f860 00000170`000000f0 00000000`0244f868 0000002c`00000001 00000000`0244f870 00000000`c0000005 00000000`0244f878 00000000`00000000 00000000`0244f880 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2 00000000`0244f888 00000000`00000002 00000000`0244f890 00000000`00000000 00000000`0244f898 00000000`00000000 00000000`0244f8a0 00000000`00000000 00000000`0244f8a8 00000000`00000000 00000000`0244f8b0 00000000`00000000 00000000`0244f8b8 00000000`00000000 00000000`0244f8c0 00000000`00000000 00000000`0244f8c8 00000000`00000000 00000000`0244f8d0 00000000`00000000 00000000`0244f8d8 00000000`00000000 00000000`0244f8e0 00000000`00000000 00000000`0244f8e8 00000000`00000000 00000000`0244f8f0 00000000`00000000 00000000`0244f8f8 00000000`00000000 00000000`0244f900 00000000`00000000 00000000`0244f908 00000000`00962210 00000000`0244f910 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2 00000000`0244f918 00000000`00000000 00000000`0244f920 00000000`00000000 00000000`0244f928 00000000`0244fab0 00000000`0244f930 00000000`77101530 ntdll!NtdllDispatchMessage_W 00000000`0244f938 00000000`76fe505b user32!DialogBox2+0×2ec 00000000`0244f940 00000000`00000000 00000000`0244f948 00000000`00000000 00000000`0244f950 00000000`00000000 00000000`0244f958 00000000`00000000 00000000`0244f960 00000000`00000000 00000000`0244f968 00000000`00000000 00000000`0244f970 00000000`00000000 00000000`0244f978 00000000`00000000 00000000`0244f980 00000000`00000002 00000000`0244f988 00000000`000111f0 00000000`0244f990 00000271`0f689359 00000000`0244f998 00000000`00000030 00000000`0244f9a0 00000000`00000000 00000000`0244f9a8 00000000`00000000 00000000`0244f9b0 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0×0) 00000000`0244f9b8 00000000`001a17e0 00000000`0244f9c0 00000000`00000000 00000000`0244f9c8 00000000`76fe4edd user32!InternalDialogBox+0×135 00000000`0244f9d0 00000000`00000000 00000000`0244f9d8 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`0244f9e0 00000000`00000000 00000000`0244f9e8 00000000`00000000 00000000`0244f9f0 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`0244f9f8 00000000`00000000 00000000`0244fa00 00000000`00000001 00000000`0244fa08 00000000`00000000 00000000`0244fa10 00000000`00000000 00000000`0244fa18 00000000`00009c40 00000000`0244fa20 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0×0) 00000000`0244fa28 00000000`76fe4f52 user32!DialogBoxIndirectParamAorW+0×58 00000000`0244fa30 00000000`001a17e0 00000000`0244fa38 00000000`00000000 00000000`0244fa40 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`0244fa48 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`0244fa50 00000000`00000000 00000000`0244fa58 00000000`00000001 00000000`0244fa60 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0×0) 00000000`0244fa68 00000000`76fdd476 user32!DialogBoxParamW+0×66 00000000`0244fa70 ffffffff`ffffffff 00000000`0244fa78 00000000`00000000 00000000`0244fa80 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc 00000000`0244fa88 00000000`00000000 00000000`0244fa90 00000000`00000000 00000000`0244fa98 00000000`00000000 00000000`0244faa0 00000000`00000000 00000000`0244faa8 00000000`ffdbdafa calc!CTimedCalc::WatchDogThread+0×72 00000000`0244fab0 00000000`00002710

Segment registers and flags look normal now:

0:003> .cxr 00000000`0244f380 rax=000000000012c770 rbx=0000000000002710 rcx=0000000000000000 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000 rip=00000000ffdbdb27 rsp=000000000244fab0 rbp=0000000000000000 r8=000000000244f938 r9=0000000000962210 r10=0000000000000000 r11=000000000244f9a0 r12=0000000000009c40 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246 calc!CTimedCalc::WatchDogThread+0xb2: 00000000`ffdbdb27 488b01 mov rax,qword ptr [rcx] ds:00000000`00000000=????????????????

0:003> k *** Stack trace for last set context - .thread/.cxr resets it Child-SP RetAddr Call Site 00000000`0244fab0 00000000`76eb59ed calc!CTimedCalc::WatchDogThread+0xb2 00000000`0244faf0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd 00000000`0244fb20 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Computer Forensics, Crash Dump Analysis, Crash Dump Patterns, Information Warfare, Memory Forensics, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 208)

Monday, June 23rd, 2014

When we suspect a particular thread doing I/O but IRP is missing in the output of !thread WinDbg command the best way is to examine the list of IRPs and associated threads from the output of !irpfind command. Here is a synthesized example from a few Virtualized Young System crash dumps:

0: kd> !thread fffffa8004e2d280

THREAD fffffa8004e2d280 Cid 0004.0020 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable fffff880009ec440 NotificationEvent Not impersonating […]

0: kd> !irpfind

Irp [ Thread ] irpStack: (Mj,Mn) DevObj [Driver] MDL Process [...] fffffa800424e4e0 [fffffa8004e2d280] irpStack: (3, 0) fffffa8004ed6d40 [ \Driver\DriverA] […]

Now we can inspect the found IRP (!irp command) and device object (for example, by using !devobj and !devstack commands). Sometimes we can see the same IRP address as Execution Residue among “Args to Child” values in the output of !thread command or kv (if the thread is current). We call such pattern Hidden IRP.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 207)

Saturday, June 21st, 2014

The pattern called Small Value deals with easily recognizable values such as handles, timeouts, mouse pointer coordinates, enumeration values, window messages, etc. There is another kind of values we call Design Values, for example, 256 (+/- 1) or some other round value. Here we can also add some regular patterns in hex representation such as window handles or flags, for example, such as 0×10008000. Such designed values may fall into some module range too, the so called Coincidental Symbolic Information pattern. If we see a design value in the output of WinDbg commands especially related to abnormal behaviour patterns, not necessarily as a stack trace parameter, which can be False, then it might point to some design limitations that were reached. For example, Blocked ALPC Queue may have a limitation on I/O Completion Port when we have ALPC Wait Chains in an unresponsive system:

0: kd> !alpc /p <port_address> [...] 512 thread(s) are registered with port IO completion object: […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Software Engineering | No Comments »

Crash Dump Analysis Patterns (Part 206)

Saturday, May 10th, 2014

Here we introduce another Wait Chain pattern where a client thread makes a request and a created server thread servicing the request makes another request to the client which creates a new client thread to service the server request. The new client thread makes a request to the server again and a new server thread is created which makes a new client request, and so on. We call such a pattern Screwbolt Wait Chain. The additional signs here may be an abnormal number of threads and possibly Handle Leak pattern although the latter may be present only in a client or server process only. Thread Age, Waiting Thread Time, and common Blocking Module patterns may be used to unwind the chain and diagnose the possible problem module and corresponding Module Product Process. The pattern is illustrated on this diagram:

Although we initially found this pattern related to LPC /ALPC IPC we think it not limited to it and can occur in different client-server communication implementations.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Pattern Prediction | No Comments »

Falsity and Coincidence Patterns

Monday, April 28th, 2014

A page to reference all different kinds of coincidence and falsity related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 205)

Saturday, April 26th, 2014

When calculating effective addresses such as [r10+10h] or [rax+rcx*12h+40h] to show their value in the output of some commands such as .trap or .cxr a debugger uses CPU register values from a saved trap frame or context structure. If such information is invalid the reported effective address doesn’t correspond to the real one during code execution. So we call this analysis pattern False Effective Address similar to False Function Parameters. Therefore, if a fault address is saved during bugcheck or exception processing it may not correspond to the output of some commands where such calculation is necessary. For example, in a bugcheck parameter we have this referenced memory address:

Arg1: fffffadda17d001d, memory referenced

but the output of .trap command shows a NULL pointer address:

NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect. rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000 [...] movzx eax,word ptr [rax+10h] 0010=????

Usually we are lucky and an effective address is correct despite the warning such as here and here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 190b)

Saturday, April 19th, 2014

While working on Thread Cluster pattern I realized that we need a predicate version of Module Collection pattern, similar to the predicate version of Stack Trace Collection pattern. A predicate can be anything: a company vendor, semantic proximity, functionality such as printing, remote file management, and so on. Such module sub-collections can be used instead of modules in more complex patterns: an example of software diagnostics pattern substitution and composition. For example, we might be able to identify a possible coupling between 2 semantically different module groups explained by IPC Wait Chains such as on this diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Software Diagnostics, Software Diagnostics Architecture, Software Diagnostics Pattern Language | No Comments »

Crash Dump Analysis Patterns (Part 204)

Friday, April 18th, 2014

One of useful patterns for the analysis of system hangs is Thread Waiting Time. If there are many such threads of interest they can be partitioned by waiting time and modules of interest from their stack traces. Modules of interest may include Directing, Coupled, Blocking, Top, and/or Problem modules depending on the problem description. We call the resulting composite pattern Thread Cluster. Extra-dimensional information can also be added such as the number of threads having the same or similar waiting time and other attributes by using different colours. For example, on this diagram, illustrating a real system hang, we see clustering of threads running through one 3rd-party module of interest and having the longest waiting time. Also we are able to identify possibly coupled (semantically related) threads running through another module of interest:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Visual Dump Analysis | 1 Comment »

Crash Dump Analysis Patterns (Part 42l)

Wednesday, April 9th, 2014

This is a variation of a general Wait Chain pattern related to CLR threads. When looking at Stack Trace Collection from a complete memory dump we may find threads using a monitor synchronization mechanism:

[... 32-bit ...] 09d2e908 6ba4d409 clr!CLREvent::WaitEx+0x106 09d2e91c 6bb90160 clr!CLREvent::Wait+0x19 09d2e9ac 6bb90256 clr!AwareLock::EnterEpilogHelper+0xa8 09d2e9ec 6bb9029b clr!AwareLock::EnterEpilog+0x42 09d2ea0c 6ba90f78 clr!AwareLock::Enter+0x5f 09d2eaa8 05952499 clr!JIT_MonEnterWorker_Portable+0xf8 […]

[... 64-bit ...] 00000000`2094e230 000007fe`eedc3e3a clr!CLREvent::WaitEx+0xc1 00000000`2094e2d0 000007fe`eedc3d43 clr!AwareLock::EnterEpilogHelper+0xca 00000000`2094e3a0 000007fe`eee3e613 clr!AwareLock::EnterEpilog+0x63 00000000`2094e400 000007ff`007f4c38 clr!JIT_MonEnterWorker_Portable+0×14f […]

When seeing such threads we may ask for a process memory dump to perform .NET memory dump analysis using SOS or other WinDbg extensions such as in Deadlock pattern example for CLR 2 (mscorwks).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | 1 Comment »

Trace Analysis Patterns (Part 76)

Sunday, December 22nd, 2013

Activity Regions or blocks of messages having the same TID or PID usually follow each other in a typical complex software trace. Such following can be completely random and independent or it may be linear based on IPC or some inter-thread communication mechanism. For example, after filtering out Background Components we my find that an RPC client call setup is followed by messages from an RPC server:

Using a coordinate approach with message number and PID axes we can reformat this minimal trace diagram:

We call such pattern Piecewise Activity where we borrowed the concept of a piecewise linear function in mathematics (and piecewise continuity). In some problem software behaviour scenarios where we encountered such analysis pattern it was complemented by Discontinuity pattern. For example, an RPC call may be blocked and we don’t see client messages after that break till the end of the trace. In such cases we always recommended forcing a complete memory dump to check for wait chain memory analysis patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in CDF Analysis Tips and Tricks, Crash Dump Analysis, Crash Dump Patterns, Log Analysis, Software Trace Analysis, Trace Analysis Patterns | 1 Comment »

Crash Dump Analysis Patterns (Part 203)

Saturday, December 7th, 2013

Sometimes we look at a stack trace collection or it’s predicate subset and recognize that one of parameters is actually the same structure address or handle. We call this pattern Shared Structure. In x64 case we may possibly see it from the return address backwards disassembly (ub WinDbg command) but in x86 case most of the time we can spot that directly from the verbose stack trace, like in the snippet below (unless a parameter memory slot was reused):

THREAD 830f9990 Cid 0428.0e94 Teb: 7ffdf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable [...] ChildEBP RetAddr Args to Child 0031f74c 7784b071 00000000 00000000 7ffdb000 ntdll!RtlpWaitOnCriticalSection+0x154 0031f774 00a91150 00a9b7a8 00000000 00a91452 ntdll!RtlEnterCriticalSection+0×152 WARNING: Stack unwind information not available. Following frames may be wrong. 0031f7c8 76113833 7ffdb000 0031f814 7784a9bd Application+0×1150 0031f7d4 7784a9bd 7ffdb000 003114bf 00000000 kernel32!BaseThreadInitThunk+0xe 0031f814 00000000 00a914a9 7ffdb000 00000000 ntdll!_RtlUserThreadStart+0×23

THREAD 886ee030 Cid 0428.0ef4 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable [...] ChildEBP RetAddr Args to Child 0098fcb8 77f881b1 00000000 00000000 001614a0 ntdll!RtlpUnWaitCriticalSection+0x1b 0098fce0 00a9102e 00a9b7a8 00000000 00000000 ntdll!RtlEnterCriticalSection+0×152 WARNING: Stack unwind information not available. Following frames may be wrong. 0098fd28 00a91275 0098fd3c 76113833 001614a0 Application+0×102e 0098fd30 76113833 001614a0 0098fd7c 7784a9bd Application+0×1275 0098fd3c 7784a9bd 001614a0 009811d7 00000000 kernel32!BaseThreadInitThunk+0xe 0098fd7c 00000000 00a911ff 001614a0 00000000 ntdll!_RtlUserThreadStart+0×23

In case of multiple exceptions or even a single exception on one thread involving invalid access to a structure field the reference to the same structure on a different thread may point to possible synchronization problems.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in ADDR Patterns, Assembly Language, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 202)

Saturday, November 9th, 2013

Sometimes we see the so called Small Values in memory (such as on raw stack) or in CPU registers which can be an ASCII or UNICODE value, some ID or even a handle. When in aggregates they can form a certain Semantic Structure such as a PID.TID example or Regular Data pattern. Here we illustrate a handle example (also an example of a Wait Chain analysis in user space):

0:000> kv Child-SP RetAddr : Args to Child : Call Site 00000000`0016de78 000007fe`fcf010dc : 00000000`02c79fa0 00000000`08c3faf0 00000000`021551f0 00000000`08c3fb00 : ntdll!NtWaitForSingleObject+0xa 00000000`0016de80 000007fe`f90e6d7f : 00000000`10b40010 00000000`10b40010 00000000`00000000 00000000`000007e0 : KERNELBASE!WaitForSingleObjectEx+0×79 […]

0:000> !handle 00000000`000007e0 ff Handle 00000000000007d0 Type Thread Attributes 0 GrantedAccess 0x1fffff: Delete,ReadControl,WriteDac,WriteOwner,Synch Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken, Impersonate,DirectImpersonate HandleCount 5 PointerCount 9 Name <none> Object specific information Thread Id 278c.a58 Priority 13 Base Priority 0

0:000> ~~[a58]s ntdll!NtWaitForMultipleObjects+0xa: 00000000`770c186a c3 ret

0:002> kv Child-SP RetAddr : Args to Child : Call Site 00000000`0f6af758 000007fe`fcf01430 : 00000000`00000025 00000000`00000000 00000000`00000000 000007fe`e35a1fb0 : ntdll!NtWaitForMultipleObjects+0xa 00000000`0f6af760 00000000`76e61220 : 00000000`0f6af8a8 00000000`0f6af890 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjectsEx+0xe8 [...]

0:026> dp 00000000`0f6af890 L4 00000000`0f6af890 00000000`00000dbc 00000000`000007c0 00000000`0f6af8a0 00000000`00000000 00000000`00000000

0:002> !handle dbc ff Handle 0000000000000dbc Type Thread Attributes 0 GrantedAccess 0x1fffff: Delete,ReadControl,WriteDac,WriteOwner,Synch Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken, Impersonate,DirectImpersonate HandleCount 2 PointerCount 4 Name <none> Object specific information Thread Id 278c.24ac Priority 14 Base Priority 0

0:002> !handle 7c0 ff Handle 00000000000007c0 Type Thread Attributes 0 GrantedAccess 0x1fffff: Delete,ReadControl,WriteDac,WriteOwner,Synch Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken, Impersonate,DirectImpersonate HandleCount 2 PointerCount 4 Name <none> Object specific information Thread Id 278c.628 Priority 14 Base Priority 0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | 1 Comment »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Archive for the ‘Crash Dump Analysis’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta