Crash Dump Analysis Patterns (Part 25d)

Some troubleshooting and debugging techniques involve saving every Stack Trace that leads to a specific action such as a memory allocation of opening of a resource handle to be saved in some region in memory, called stack trace database. Typical pattern usage examples include Process Heap Memory Leak, Insufficient Memory due to Handle Leak. Typical entry in such a database consists of return addresses saved during function calls (which may be Truncated Stack Trace):

00000000`00325da0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa
00000000`00325da8 00000001`3fd72239 AllocFree!_ioinit+0×2cd
00000000`00325db0 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5
00000000`00325db8 00000000`773759ed kernel32!BaseThreadInitThunk+0xd
00000000`00325dc0 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d

0:001> ub 00000001`3fd72239
AllocFree!_ioinit+0×2af:
00000001`3fd7221b cmp eax,3
00000001`3fd7221e jne AllocFree!_ioinit+0×2be (00000001`3fd7222a)
00000001`3fd72220 movsx eax,byte ptr [rbx+8]
00000001`3fd72224 or eax,8
00000001`3fd72227 mov byte ptr [rbx+8],al
00000001`3fd7222a lea rcx,[rbx+10h]
00000001`3fd7222e mov edx,0FA0h
00000001`3fd72233 call qword ptr [AllocFree!_imp_InitializeCriticalSectionAndSpinCount (00000001`3fd78090)

This slightly differs from ‘k’-style stack trace format where the return address belongs to the function on the next line if moving downwards:

0:000> k
Child-SP RetAddr Call Site
00000000`002ff9f8 000007fe`fd5e1203 ntdll!ZwDelayExecution+0xa
00000000`002ffa00 00000001`3fd71018 KERNELBASE!SleepEx+0xab
00000000`002ffaa0 00000001`3fd71194 AllocFree!wmain+0×18
00000000`002ffad0 00000000`773759ed AllocFree!__tmainCRTStartup+0×144
00000000`002ffb10 00000000`774ac541 kernel32!BaseThreadInitThunk+0xd
00000000`002ffb40 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub 00000001`3fd71194
AllocFree!__tmainCRTStartup+0×11b:
00000001`3fd7116b je AllocFree!__tmainCRTStartup+0×124 (00000001`3fd71174)
00000001`3fd7116d mov ecx,eax
00000001`3fd7116f call AllocFree!_amsg_exit (00000001`3fd718ec)
00000001`3fd71174 mov r8,qword ptr [AllocFree!_wenviron (00000001`3fd80868)]
00000001`3fd7117b mov qword ptr [AllocFree!__winitenv (00000001`3fd80890)],r8
00000001`3fd71182 mov rdx,qword ptr [AllocFree!__wargv (00000001`3fd80858)]
00000001`3fd71189 mov ecx,dword ptr [AllocFree!__argc (00000001`3fd8084c)]
00000001`3fd7118f call AllocFree!wmain (00000001`3fd71000)

Sometimes we can see such traces as Execution Residue inside a stack or some other region. If user mode stack trace database is enabled in gflags.exe we might be able to dump the specific database region:

0:001> !gflag
Current NtGlobalFlag contents: 0x00001000
ust - Create user mode stack trace database

0:001> !address
[...]
BaseAddress  EndAddress+1 RegionSize Type        State       Protect        Usage
------------------------------------------------------------------------------------------------------------------------
[...]
+ 0`00300000 0`00326000   0`00026000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE Other [Stack Trace Database]
0`00326000 0`01aff000   0`017d9000 MEM_PRIVATE MEM_RESERVE                Other [Stack Trace Database]
0`01aff000 0`01b00000   0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE Other [Stack Trace Database]
[…]

0:001> dps 0`00326000-1000 0`00326000
[…]
00000000`003257e0 00000000`00000000
00000000`003257e8 00030001`00001801
00000000`003257f0 00000000`774c34eb ntdll!LdrpInitializeProcess+0×7e6
00000000`003257f8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325800 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325808 00000000`00000000
00000000`00325810 00000000`00000000
00000000`00325818 00030002`00001801
00000000`00325820 00000000`774c3511 ntdll!LdrpInitializeProcess+0×80c
00000000`00325828 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325830 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325838 00000000`00000000
00000000`00325840 00000000`00000000
00000000`00325848 00040003`00001801
00000000`00325850 00000000`774bda86 ntdll!RtlCreateHeap+0×506
00000000`00325858 00000000`774c3557 ntdll!LdrpInitializeProcess+0×851
00000000`00325860 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325868 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325870 00000000`00000000
00000000`00325878 00050004`00002801
00000000`00325880 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a
00000000`00325888 00000000`774bdaee ntdll!RtlCreateHeap+0×56e
00000000`00325890 00000000`774c3557 ntdll!LdrpInitializeProcess+0×851
00000000`00325898 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`003258a0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`003258a8 00000000`00000000
00000000`003258b0 00000000`00000000
00000000`003258b8 00030005`00001801
00000000`003258c0 00000000`774c359e ntdll!LdrpInitializeProcess+0×902
00000000`003258c8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`003258d0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`003258d8 00000000`00000000
00000000`003258e0 00000000`00000000
00000000`003258e8 00030006`00001801
00000000`003258f0 00000000`774c35af ntdll!LdrpInitializeProcess+0×913
00000000`003258f8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325900 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325908 00000000`00000000
00000000`00325910 00000000`00000000
00000000`00325918 00090007`00004801
00000000`00325920 00000000`774bda86 ntdll!RtlCreateHeap+0×506
00000000`00325928 00000000`774c47ff ntdll!CsrpConnectToServer+0×41f
00000000`00325930 00000000`774c43c5 ntdll!CsrClientConnectToServer+0×230
00000000`00325938 000007fe`fd5ee232 KERNELBASE!KernelBaseDllInitialize+0×148
00000000`00325940 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325948 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325950 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325958 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325960 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325968 00000000`00000000
00000000`00325970 00000000`00000000
00000000`00325978 000a0008`00004801
00000000`00325980 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a
00000000`00325988 00000000`774bdaee ntdll!RtlCreateHeap+0×56e
00000000`00325990 00000000`774c47ff ntdll!CsrpConnectToServer+0×41f
00000000`00325998 00000000`774c43c5 ntdll!CsrClientConnectToServer+0×230
00000000`003259a0 000007fe`fd5ee232 KERNELBASE!KernelBaseDllInitialize+0×148
00000000`003259a8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`003259b0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`003259b8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`003259c0 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`003259c8 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`003259d0 00000000`00000000
00000000`003259d8 00080009`00003801
00000000`003259e0 000007fe`fd5edf81 KERNELBASE!NlsProcessInitialize+0×11
00000000`003259e8 000007fe`fd604439 KERNELBASE!BaseNlsDllInitialize+0×29
00000000`003259f0 000007fe`fd5ee446 KERNELBASE!KernelBaseDllInitialize+0×40c
00000000`003259f8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325a00 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325a08 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325a10 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325a18 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325a20 00000000`00000000
00000000`00325a28 0008000a`00003801
00000000`00325a30 000007fe`fd5edfa0 KERNELBASE!NlsProcessInitialize+0×30
00000000`00325a38 000007fe`fd604439 KERNELBASE!BaseNlsDllInitialize+0×29
00000000`00325a40 000007fe`fd5ee446 KERNELBASE!KernelBaseDllInitialize+0×40c
00000000`00325a48 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325a50 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325a58 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325a60 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325a68 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325a70 00000000`00000000
00000000`00325a78 0007000b`00003801
00000000`00325a80 000007fe`fd604a21 KERNELBASE!BasepInitComputerNameCache+0×11
00000000`00325a88 000007fe`fd603d20 KERNELBASE!KernelBaseDllInitialize+0×419
00000000`00325a90 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325a98 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325aa0 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325aa8 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325ab0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325ab8 00000000`00000000
00000000`00325ac0 00000000`00000000
00000000`00325ac8 0006000c`00002801
00000000`00325ad0 00000000`77375699 kernel32!BaseDllInitialize+0×2f9
00000000`00325ad8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325ae0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325ae8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325af0 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325af8 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325b00 00000000`00000000
00000000`00325b08 0007000d`00003801
00000000`00325b10 00000000`773771f7 kernel32!InitializeConsoleConnectionInfo+0xe7
00000000`00325b18 00000000`773756ae kernel32!BaseDllInitialize+0×30e
00000000`00325b20 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325b28 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325b30 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325b38 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325b40 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325b48 00000000`00000000
00000000`00325b50 00000000`00000000
00000000`00325b58 0009000e`00004801
00000000`00325b60 00000000`774bda86 ntdll!RtlCreateHeap+0×506
00000000`00325b68 00000000`773787f7 kernel32!ConsoleConnect+0×1d7
00000000`00325b70 00000000`773770de kernel32!ConnectConsoleInternal+0×147
00000000`00325b78 00000000`773756fe kernel32!BaseDllInitialize+0×35e
00000000`00325b80 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325b88 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325b90 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325b98 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325ba0 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325ba8 00000000`00000000
00000000`00325bb0 00000000`00000000
00000000`00325bb8 000a000f`00004801
00000000`00325bc0 00000000`7751998a ntdll! ?? ::FNODOBFM::`string’+0xdc1a
00000000`00325bc8 00000000`774bdaee ntdll!RtlCreateHeap+0×56e
00000000`00325bd0 00000000`773787f7 kernel32!ConsoleConnect+0×1d7
00000000`00325bd8 00000000`773770de kernel32!ConnectConsoleInternal+0×147
00000000`00325be0 00000000`773756fe kernel32!BaseDllInitialize+0×35e
00000000`00325be8 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325bf0 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325bf8 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325c00 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325c08 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325c10 00000000`00000000
00000000`00325c18 00060010`00002801
00000000`00325c20 00000000`773757dc kernel32!BaseDllInitialize+0×43c
00000000`00325c28 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325c30 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325c38 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325c40 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325c48 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325c50 00000000`00000000
00000000`00325c58 00060011`00002801
00000000`00325c60 00000000`7737582c kernel32!BaseDllInitialize+0×48c
00000000`00325c68 00000000`774bb108 ntdll!LdrpRunInitializeRoutines+0×1fe
00000000`00325c70 00000000`774c42fd ntdll!LdrGetProcedureAddressEx+0×2aa
00000000`00325c78 00000000`774c1ddc ntdll!LdrpInitializeProcess+0×1a0b
00000000`00325c80 00000000`774c1937 ntdll! ?? ::FNODOBFM::`string’+0×28ff0
00000000`00325c88 00000000`774ac34e ntdll!LdrInitializeThunk+0xe
00000000`00325c90 00000000`00000000
00000000`00325c98 00060012`0000280e
00000000`00325ca0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa
00000000`00325ca8 00000001`3fd7319f AllocFree!_mtinitlocks+0×43
00000000`00325cb0 00000001`3fd717fc AllocFree!_mtinit+0×10
00000000`00325cb8 00000001`3fd710e4 AllocFree!__tmainCRTStartup+0×94
00000000`00325cc0 00000000`773759ed kernel32!BaseThreadInitThunk+0xd
00000000`00325cc8 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d
00000000`00325cd0 00000000`00000000
00000000`00325cd8 000b0013`00005801
00000000`00325ce0 00000000`774c1131 ntdll!RtlpActivateLowFragmentationHeap+0×181
00000000`00325ce8 00000000`774c0f97 ntdll!RtlpPerformHeapMaintenance+0×27
00000000`00325cf0 00000000`774c0f5b ntdll!RtlpAllocateHeap+0×1819
00000000`00325cf8 00000000`774d34d8 ntdll!RtlAllocateHeap+0×16c
00000000`00325d00 00000000`774a9300 ntdll!RtlInitializeCriticalSectionAndSpinCount+0×183
00000000`00325d08 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa
00000000`00325d10 00000001`3fd7319f AllocFree!_mtinitlocks+0×43
00000000`00325d18 00000001`3fd717fc AllocFree!_mtinit+0×10
00000000`00325d20 00000001`3fd710e4 AllocFree!__tmainCRTStartup+0×94
00000000`00325d28 00000000`773759ed kernel32!BaseThreadInitThunk+0xd
00000000`00325d30 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d
00000000`00325d38 00000000`00000000
00000000`00325d40 00000000`00000000
00000000`00325d48 00070014`00003801
00000000`00325d50 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa
00000000`00325d58 00000001`3fd7312f AllocFree!_mtinitlocknum+0×8f
00000000`00325d60 00000001`3fd72ff7 AllocFree!_lock+0×23
00000000`00325d68 00000001`3fd71f9b AllocFree!_ioinit+0×2f
00000000`00325d70 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5
00000000`00325d78 00000000`773759ed kernel32!BaseThreadInitThunk+0xd
00000000`00325d80 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d
00000000`00325d88 00000000`00000000
00000000`00325d90 00000000`00000000
00000000`00325d98 00050015`00002803
00000000`00325da0 000007fe`fd5e37aa KERNELBASE!InitializeCriticalSectionAndSpinCount+0xa
00000000`00325da8 00000001`3fd72239 AllocFree!_ioinit+0×2cd
00000000`00325db0 00000001`3fd71115 AllocFree!__tmainCRTStartup+0xc5
00000000`00325db8 00000000`773759ed kernel32!BaseThreadInitThunk+0xd
00000000`00325dc0 00000000`774ac541 ntdll!RtlUserThreadStart+0×1d
00000000`00325dc8 00000000`00000000
00000000`00325dd0 00000000`00000000
[…]

This database corresponds to this simple program:

int _tmain(int argc, _TCHAR* argv[])
{
    free(malloc(256));
    Sleep(-1);
    return 0;
}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply