Comments on: Crash Dump Analysis Patterns (Part 133) https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/ Structural and Behavioral Patterns for Software Diagnostics, Forensics and Prognostics Thu, 07 May 2026 00:19:31 +0000 http://wordpress.org/?v=2.3.3 By: Dmitry Vostokov https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-741723 Dmitry Vostokov Sun, 23 Oct 2016 08:47:48 +0000 https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-741723 Sometimes WinDbg commands show only limited number of frames. For example, !cs -l -o -s command may not show heap corruption as the cause of heap-loader deadlock if unhandled exception processing is too long. We need to double check stack traces with k commands. See also: http://www.dumpanalysis.org/blog/index.php/2008/05/27/10-common-mistakes-in-memory-analysis-part-1/ Sometimes WinDbg commands show only limited number of frames. For example, !cs -l -o -s command may not show heap corruption as the cause of heap-loader deadlock if unhandled exception processing is too long. We need to double check stack traces with k commands. See also: http://www.dumpanalysis.org/blog/index.php/2008/05/27/10-common-mistakes-in-memory-analysis-part-1/

]]> By: Dmitry Vostokov https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-741714 Dmitry Vostokov Wed, 14 Sep 2016 15:17:25 +0000 https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-741714 Some truncated stack traces originate from JIT Code: 00007ffa86d9952c: ntdll!RtlAllocateHeap+0x00000000000788fc 0000000058248d17: msvcr100!malloc+0x000000000000005b 0000000058248ddb: msvcr100!operator new+0x000000000000001f 00007ffa1f1a77f8: +0x00007ffa1f1a77f8 0:000> ub 0x00007ffa1f1a77f8 00007ffa`1f1a77d5 488b4320 mov rax,qword ptr [rbx+20h] 00007ffa`1f1a77d9 48895d18 mov qword ptr [rbp+18h],rbx 00007ffa`1f1a77dd 488d1514000000 lea rdx,[00007ffa`1f1a77f8] 00007ffa`1f1a77e4 48895530 mov qword ptr [rbp+30h],rdx 00007ffa`1f1a77e8 c7410c00000000 mov dword ptr [rcx+0Ch],0 00007ffa`1f1a77ef 488b4d40 mov rcx,qword ptr [rbp+40h] 00007ffa`1f1a77f3 4533db xor r11d,r11d 00007ffa`1f1a77f6 ff10 call qword ptr [rax] 0:000> !IP2MD 0x00007ffa1f1a77f8 MethodDesc: 00007ffa1f292678 Method Name: ModuleA.Foo(UInt64) Class: 00007ffa1f204b90 MethodTable: 00007ffa1f204c08 mdToken: 0000000006000000 Module: 00007ffa1f203760 IsJitted: yes CodeAddr: 00007ffa1f1a7790 Transparency: Safe critical *** ERROR: Module load completed but symbols could not be loaded for Company.Subsystem.Component.dll Some truncated stack traces originate from JIT Code:

00007ffa86d9952c: ntdll!RtlAllocateHeap+0×00000000000788fc
0000000058248d17: msvcr100!malloc+0×000000000000005b
0000000058248ddb: msvcr100!operator new+0×000000000000001f
00007ffa1f1a77f8: +0×00007ffa1f1a77f8

0:000> ub 0×00007ffa1f1a77f8
00007ffa`1f1a77d5 488b4320 mov rax,qword ptr [rbx+20h]
00007ffa`1f1a77d9 48895d18 mov qword ptr [rbp+18h],rbx
00007ffa`1f1a77dd 488d1514000000 lea rdx,[00007ffa`1f1a77f8]
00007ffa`1f1a77e4 48895530 mov qword ptr [rbp+30h],rdx
00007ffa`1f1a77e8 c7410c00000000 mov dword ptr [rcx+0Ch],0
00007ffa`1f1a77ef 488b4d40 mov rcx,qword ptr [rbp+40h]
00007ffa`1f1a77f3 4533db xor r11d,r11d
00007ffa`1f1a77f6 ff10 call qword ptr [rax]

0:000> !IP2MD 0×00007ffa1f1a77f8
MethodDesc: 00007ffa1f292678
Method Name: ModuleA.Foo(UInt64)
Class: 00007ffa1f204b90
MethodTable: 00007ffa1f204c08
mdToken: 0000000006000000
Module: 00007ffa1f203760
IsJitted: yes
CodeAddr: 00007ffa1f1a7790
Transparency: Safe critical
*** ERROR: Module load completed but symbols could not be loaded for Company.Subsystem.Component.dll

]]> By: Dmitry Vostokov https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-741701 Dmitry Vostokov Tue, 03 May 2016 21:27:37 +0000 https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-741701 <p align="left">Here's another example of Truncated Stack Trace from x64 Windows 10 after NULL code pointer was called. <p align="left">0:003> kL # Child-SP RetAddr Call Site 00 0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa 01 0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef 02 0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe 03 0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0x494 04 0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0x48 05 0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0x1fd 06 0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0x3e 07 0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0x96 08 0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd 09 0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0x370 0a 0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0x3a <p align="left">Here we can look at the trap frame RSP address and recover the return address: <p align="left">0:003> dt _KTRAP_FRAME RSP 0000005b`056af7e8 ntdll!_KTRAP_FRAME +0x180 Rsp : 0x0000005b`056afad8 <p align="left">0:003> dps 0x0000005b`056afad8 L1 0000005b`056afad8 00007ff7`f6381266 ApplicationM!thread_two+0x16 <p align="left">0:003> ub 00007ff7`f6381266 [...] ApplicationM!thread_two: 00007ff7`f6381250 48894c2408 mov qword ptr [rsp+8],rcx 00007ff7`f6381255 4883ec38 sub rsp,38h 00007ff7`f6381259 48c744242000000000 mov qword ptr [rsp+20h],0 00007ff7`f6381262 ff542420 call qword ptr [rsp+20h] <p align="left">and reconstruct the bottom of the stack trace: <p align="left">0:003> k L=0x0000005b`056afad8 # Child-SP RetAddr Call Site 00 0000005b`056afad8 00007ff7`f6381266 ntdll!NtWaitForMultipleObjects+0xa 01 0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0x16 02 (Inline Function) --------`-------- ApplicationM!invoke_thread_procedure+0xe 03 0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start<void (__cdecl*)(void * __ptr64)>+0x5d 04 0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0x22 05 0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0x34 <p align="left">and then construct Glued Stack Trace: <p align="left">0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa 0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef 0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe 0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0x494 0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0x48 0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0x1fd 0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0x3e 0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0x96 0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd 0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0x370 0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0x3a 0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0x16 (Inline Function) --------`-------- ApplicationM!invoke_thread_procedure+0xe 0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start<void (__cdecl*)(void * __ptr64)>+0x5d 0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0x22 0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0x34 Here’s another example of Truncated Stack Trace from x64 Windows 10 after NULL code pointer was called.

0:003> kL
# Child-SP RetAddr Call Site
00 0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa
01 0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef
02 0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe
03 0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0×494
04 0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0×48
05 0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0×1fd
06 0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0×3e
07 0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0×96
08 0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd
09 0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0×370
0a 0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0×3a

Here we can look at the trap frame RSP address and recover the return address:

0:003> dt _KTRAP_FRAME RSP 0000005b`056af7e8
ntdll!_KTRAP_FRAME
+0×180 Rsp : 0×0000005b`056afad8

0:003> dps 0×0000005b`056afad8 L1
0000005b`056afad8 00007ff7`f6381266 ApplicationM!thread_two+0×16

0:003> ub 00007ff7`f6381266
[…]
ApplicationM!thread_two:
00007ff7`f6381250 48894c2408 mov qword ptr [rsp+8],rcx
00007ff7`f6381255 4883ec38 sub rsp,38h
00007ff7`f6381259 48c744242000000000 mov qword ptr [rsp+20h],0
00007ff7`f6381262 ff542420 call qword ptr [rsp+20h]

and reconstruct the bottom of the stack trace:

0:003> k L=0×0000005b`056afad8
# Child-SP RetAddr Call Site
00 0000005b`056afad8 00007ff7`f6381266 ntdll!NtWaitForMultipleObjects+0xa
01 0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0×16
02 (Inline Function) ——–`——– ApplicationM!invoke_thread_procedure+0xe
03 0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start+0×5d
04 0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0×22
05 0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0×34

and then construct Glued Stack Trace:

0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa
0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef
0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe
0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0×494
0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0×48
0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0×1fd
0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0×3e
0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0×96
0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd
0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0×370
0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0×3a
0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0×16
(Inline Function) ——–`——– ApplicationM!invoke_thread_procedure+0xe
0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start+0×5d
0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0×22
0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0×34

]]> By: Dmitry Vostokov https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-349574 Dmitry Vostokov Mon, 03 Oct 2011 14:52:14 +0000 https://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133/#comment-349574 Another example is when a debugger truncates the output. In WinDbg case there is a command .kframes that sets the default Another example is when a debugger truncates the output. In WinDbg case there is a command .kframes that sets the default

]]>