« Analysis Compass
Resuming Literate Scientist Blog »

Crash Dump Analysis Patterns (Part 133)

Sometimes we see Truncated Stack Trace with missing stack frames. For example, in one incident, after enabling user mode stack trace database for a memory leaking application we got these entries from the growing heap segment (other segments had non-truncated saved stack traces):

0bdc1350: 40010 . 40010 [101] - busy (3fff8) Internal

    7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
    77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
    77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
    7702661f: ntdll!RtlAllocateHeap+0x0000017c

0be01360: 40010 . 40010 [101] - busy (3fff8) Internal

    7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
    77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
    77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
    7702661f: ntdll!RtlAllocateHeap+0x0000017c

0be41370: 40010 . 40010 [101] - busy (3fff8) Internal

    7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
    77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
    77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
    7702661f: ntdll!RtlAllocateHeap+0x0000017c

Truncated traces are different from incorrect stack traces because their surviving part is correct. How can we find the rest of such stack traces? Here we can suggest to look at other heap segments and see allocations of the same size. If a truncated trace comes from a stack trace collection we can compare it with a non-truncated thread stack from another process instance having the same thread position. Other suggestions are welcome too, especially, for the mysterious memory leak shown above.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This entry was posted on Sunday, March 20th, 2011 at 12:06 am and is filed under Crash Dump Analysis, Crash Dump Patterns, Debugging. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

4 Responses to “Crash Dump Analysis Patterns (Part 133)”

  1. Dmitry Vostokov Says:
    October 3rd, 2011 at 2:52 pm

    Another example is when a debugger truncates the output. In WinDbg case there is a command .kframes that sets the default

  2. Dmitry Vostokov Says:
    May 3rd, 2016 at 9:27 pm

    Here’s another example of Truncated Stack Trace from x64 Windows 10 after NULL code pointer was called.

    0:003> kL
    # Child-SP RetAddr Call Site
    00 0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa
    01 0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef
    02 0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe
    03 0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0×494
    04 0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0×48
    05 0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0×1fd
    06 0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0×3e
    07 0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0×96
    08 0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd
    09 0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0×370
    0a 0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0×3a

    Here we can look at the trap frame RSP address and recover the return address:

    0:003> dt _KTRAP_FRAME RSP 0000005b`056af7e8
    ntdll!_KTRAP_FRAME
    +0×180 Rsp : 0×0000005b`056afad8

    0:003> dps 0×0000005b`056afad8 L1
    0000005b`056afad8 00007ff7`f6381266 ApplicationM!thread_two+0×16

    0:003> ub 00007ff7`f6381266
    […]
    ApplicationM!thread_two:
    00007ff7`f6381250 48894c2408 mov qword ptr [rsp+8],rcx
    00007ff7`f6381255 4883ec38 sub rsp,38h
    00007ff7`f6381259 48c744242000000000 mov qword ptr [rsp+20h],0
    00007ff7`f6381262 ff542420 call qword ptr [rsp+20h]

    and reconstruct the bottom of the stack trace:

    0:003> k L=0×0000005b`056afad8
    # Child-SP RetAddr Call Site
    00 0000005b`056afad8 00007ff7`f6381266 ntdll!NtWaitForMultipleObjects+0xa
    01 0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0×16
    02 (Inline Function) ——–`——– ApplicationM!invoke_thread_procedure+0xe
    03 0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start+0×5d
    04 0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0×22
    05 0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0×34

    and then construct Glued Stack Trace:

    0000005b`056ae1f8 00007ffa`e2f2916f ntdll!NtWaitForMultipleObjects+0xa
    0000005b`056ae200 00007ffa`e2f2906e KERNELBASE!WaitForMultipleObjectsEx+0xef
    0000005b`056ae500 00007ffa`e3b5155c KERNELBASE!WaitForMultipleObjects+0xe
    0000005b`056ae540 00007ffa`e3b51088 kernel32!WerpReportFaultInternal+0×494
    0000005b`056aeab0 00007ffa`e2f503ad kernel32!WerpReportFault+0×48
    0000005b`056aeae0 00007ffa`e5c58cd2 KERNELBASE!UnhandledExceptionFilter+0×1fd
    0000005b`056aebe0 00007ffa`e5c44296 ntdll!RtlUserThreadStart$filt$0+0×3e
    0000005b`056aec20 00007ffa`e5c5666d ntdll!_C_specific_handler+0×96
    0000005b`056aec90 00007ffa`e5bd3c00 ntdll!RtlpExecuteHandlerForException+0xd
    0000005b`056aecc0 00007ffa`e5c5577a ntdll!RtlDispatchException+0×370
    0000005b`056af3c0 00000000`00000000 ntdll!KiUserExceptionDispatch+0×3a
    0000005b`056afae0 00007ff7`f6382bd1 ApplicationM!thread_two+0×16
    (Inline Function) ——–`——– ApplicationM!invoke_thread_procedure+0xe
    0000005b`056afb20 00007ffa`e3b42d92 ApplicationM!thread_start+0×5d
    0000005b`056afb50 00007ffa`e5bc9f64 kernel32!BaseThreadInitThunk+0×22
    0000005b`056afb80 00000000`00000000 ntdll!RtlUserThreadStart+0×34

  3. Dmitry Vostokov Says:
    September 14th, 2016 at 3:17 pm

    Some truncated stack traces originate from JIT Code:

    00007ffa86d9952c: ntdll!RtlAllocateHeap+0×00000000000788fc
    0000000058248d17: msvcr100!malloc+0×000000000000005b
    0000000058248ddb: msvcr100!operator new+0×000000000000001f
    00007ffa1f1a77f8: +0×00007ffa1f1a77f8

    0:000> ub 0×00007ffa1f1a77f8
    00007ffa`1f1a77d5 488b4320 mov rax,qword ptr [rbx+20h]
    00007ffa`1f1a77d9 48895d18 mov qword ptr [rbp+18h],rbx
    00007ffa`1f1a77dd 488d1514000000 lea rdx,[00007ffa`1f1a77f8]
    00007ffa`1f1a77e4 48895530 mov qword ptr [rbp+30h],rdx
    00007ffa`1f1a77e8 c7410c00000000 mov dword ptr [rcx+0Ch],0
    00007ffa`1f1a77ef 488b4d40 mov rcx,qword ptr [rbp+40h]
    00007ffa`1f1a77f3 4533db xor r11d,r11d
    00007ffa`1f1a77f6 ff10 call qword ptr [rax]

    0:000> !IP2MD 0×00007ffa1f1a77f8
    MethodDesc: 00007ffa1f292678
    Method Name: ModuleA.Foo(UInt64)
    Class: 00007ffa1f204b90
    MethodTable: 00007ffa1f204c08
    mdToken: 0000000006000000
    Module: 00007ffa1f203760
    IsJitted: yes
    CodeAddr: 00007ffa1f1a7790
    Transparency: Safe critical
    *** ERROR: Module load completed but symbols could not be loaded for Company.Subsystem.Component.dll

  4. Dmitry Vostokov Says:
    October 23rd, 2016 at 8:47 am

    Sometimes WinDbg commands show only limited number of frames. For example, !cs -l -o -s command may not show heap corruption as the cause of heap-loader deadlock if unhandled exception processing is too long. We need to double check stack traces with k commands. See also: http://www.dumpanalysis.org/blog/index.php/2008/05/27/10-common-mistakes-in-memory-analysis-part-1/

Leave a Reply

You must be logged in to post a comment.