Crash Dump Analysis Patterns (Part 133)
Sometimes we see Truncated Stack Trace with missing stack frames. For example, in one incident, after enabling user mode stack trace database for a memory leaking application we got these entries from the growing heap segment (other segments had non-truncated saved stack traces):
0bdc1350: 40010 . 40010 [101] - busy (3fff8) Internal
7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
7702661f: ntdll!RtlAllocateHeap+0x0000017c
0be01360: 40010 . 40010 [101] - busy (3fff8) Internal
7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
7702661f: ntdll!RtlAllocateHeap+0x0000017c
0be41370: 40010 . 40010 [101] - busy (3fff8) Internal
7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
7702661f: ntdll!RtlAllocateHeap+0x0000017c
Truncated traces are different from incorrect stack traces because their surviving part is correct. How can we find the rest of such stack traces? Here we can suggest to look at other heap segments and see allocations of the same size. If a truncated trace comes from a stack trace collection we can compare it with a non-truncated thread stack from another process instance having the same thread position. Other suggestions are welcome too, especially, for the mysterious memory leak shown above.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
October 3rd, 2011 at 2:52 pm
Another example is when a debugger truncates the output. In WinDbg case there is a command .kframes that sets the default
May 3rd, 2016 at 9:27 pm
Here’s another example of Truncated Stack Trace from x64 Windows 10 after NULL code pointer was called.
0:003> kL
# Child-SP RetAddr Call Site
00 0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa
01 0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef
02 0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe
03 0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0×494
04 0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0×48
05 0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0×1fd
06 0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0×3e
07 0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0×96
08 0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd
09 0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0×370
0a 0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0×3a
Here we can look at the trap frame RSP address and recover the return address:
0:003> dt _KTRAP_FRAME RSP 0000005b`056af7e8
ntdll!_KTRAP_FRAME
+0×180 Rsp : 0×0000005b`056afad8
0:003> dps 0×0000005b`056afad8 L1
0000005b`056afad8 00007ff7`f6381266 ApplicationM!thread_two+0×16
0:003> ub 00007ff7`f6381266
[…]
ApplicationM!thread_two:
00007ff7`f6381250 48894c2408 mov qword ptr [rsp+8],rcx
00007ff7`f6381255 4883ec38 sub rsp,38h
00007ff7`f6381259 48c744242000000000 mov qword ptr [rsp+20h],0
00007ff7`f6381262 ff542420 call qword ptr [rsp+20h]
and reconstruct the bottom of the stack trace:
0:003> k L=0×0000005b`056afad8+0×5d
# Child-SP RetAddr Call Site
00 0000005b`056afad8 00007ff7`f6381266 ntdll!NtWaitForMultipleObjects+0xa
01 0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0×16
02 (Inline Function) ——–`——– ApplicationM!invoke_thread_procedure+0xe
03 0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start
04 0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0×22
05 0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0×34
and then construct Glued Stack Trace:
0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa+0×5d
0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef
0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe
0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0×494
0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0×48
0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0×1fd
0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0×3e
0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0×96
0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd
0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0×370
0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0×3a
0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0×16
(Inline Function) ——–`——– ApplicationM!invoke_thread_procedure+0xe
0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start
0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0×22
0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0×34
September 14th, 2016 at 3:17 pm
Some truncated stack traces originate from JIT Code:
00007ffa86d9952c: ntdll!RtlAllocateHeap+0×00000000000788fc
0000000058248d17: msvcr100!malloc+0×000000000000005b
0000000058248ddb: msvcr100!operator new+0×000000000000001f
00007ffa1f1a77f8: +0×00007ffa1f1a77f8
0:000> ub 0×00007ffa1f1a77f8
00007ffa`1f1a77d5 488b4320 mov rax,qword ptr [rbx+20h]
00007ffa`1f1a77d9 48895d18 mov qword ptr [rbp+18h],rbx
00007ffa`1f1a77dd 488d1514000000 lea rdx,[00007ffa`1f1a77f8]
00007ffa`1f1a77e4 48895530 mov qword ptr [rbp+30h],rdx
00007ffa`1f1a77e8 c7410c00000000 mov dword ptr [rcx+0Ch],0
00007ffa`1f1a77ef 488b4d40 mov rcx,qword ptr [rbp+40h]
00007ffa`1f1a77f3 4533db xor r11d,r11d
00007ffa`1f1a77f6 ff10 call qword ptr [rax]
0:000> !IP2MD 0×00007ffa1f1a77f8
MethodDesc: 00007ffa1f292678
Method Name: ModuleA.Foo(UInt64)
Class: 00007ffa1f204b90
MethodTable: 00007ffa1f204c08
mdToken: 0000000006000000
Module: 00007ffa1f203760
IsJitted: yes
CodeAddr: 00007ffa1f1a7790
Transparency: Safe critical
*** ERROR: Module load completed but symbols could not be loaded for Company.Subsystem.Component.dll
October 23rd, 2016 at 8:47 am
Sometimes WinDbg commands show only limited number of frames. For example, !cs -l -o -s command may not show heap corruption as the cause of heap-loader deadlock if unhandled exception processing is too long. We need to double check stack traces with k commands. See also: http://www.dumpanalysis.org/blog/index.php/2008/05/27/10-common-mistakes-in-memory-analysis-part-1/