Archive for November, 2019

Crash Dump Analysis Patterns (Part 262)

Saturday, November 30th, 2019

Full symbol files usually include source code information for UI debugging. Debuggers, like WinDbg, may show it for each relevant stack trace frame:

0:001> k
# Child-SP          RetAddr           Call Site
00 00000074`fe5fe3c8 00007ff9`72937ff7 ntdll!NtWaitForMultipleObjects+0x14
01 00000074`fe5fe3d0 00007ff9`72937ede KERNELBASE!WaitForMultipleObjectsEx+0x107
02 00000074`fe5fe6d0 00007ff9`74a671fb KERNELBASE!WaitForMultipleObjects+0xe
03 00000074`fe5fe710 00007ff9`74a66ca8 kernel32!WerpReportFaultInternal+0x51b
04 00000074`fe5fe830 00007ff9`729df848 kernel32!WerpReportFault+0xac
05 00000074`fe5fe870 00007ff9`75304af2 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 00000074`fe5fe990 00007ff9`752ec6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
07 00000074`fe5fe9d0 00007ff9`753011ff ntdll!_C_specific_handler+0x96
08 00000074`fe5fea40 00007ff9`752ca289 ntdll!RtlpExecuteHandlerForException+0xf
09 00000074`fe5fea70 00007ff9`752ffe6e ntdll!RtlDispatchException+0x219
0a 00000074`fe5ff180 00007ff7`bbd51cb0 ntdll!KiUserExceptionDispatch+0x2e
0b 00000074`fe5ff930 00007ff7`bbd51983 SourceStackTrace!std::_Deque_unchecked_const_iterator<std::_Deque_val< std::_Deque_simple_types<int> > >::operator*+0x50 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\deque @ 55]
0c 00000074`fe5ff970 00007ff7`bbd51883 SourceStackTrace!std::_Deque_unchecked_iterator<std::_Deque_val< std::_Deque_simple_types<int> > >::operator*+0×13 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\deque @ 175]
0d 00000074`fe5ff9a0 00007ff7`bbd515f6 SourceStackTrace!std::deque<int,std::allocator<int> >::back+0×33 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\deque @ 1091]
0e 00000074`fe5ff9f0 00007ff7`bbd5156c SourceStackTrace!std::stack<int,std::deque<int,std::allocator<int> > >::top+0×16 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\stack @ 66]
0f 00000074`fe5ffa20 00007ff7`bbd521a3 SourceStackTrace!Info::gather+0×3c [C:\NewWork\SourceStackTrace\InfoGather.cpp @ 9]
10 00000074`fe5ffa90 00007ff7`bbd5108d SourceStackTrace!Info::scatter+0×13 [C:\NewWork\SourceStackTrace\InfoScatter.cpp @ 6]
11 00000074`fe5ffac0 00007ff7`bbd510a9 SourceStackTrace!worker_bar+0×1d [C:\NewWork\SourceStackTrace\bar.cpp @ 10]
12 00000074`fe5ffb00 00007ff7`bbd522b9 SourceStackTrace!worker_foo+0×9 [C:\NewWork\SourceStackTrace\foo.cpp @ 6]
13 00000074`fe5ffb30 00007ff7`bbd526b5 SourceStackTrace!worker+0×9 [C:\NewWork\SourceStackTrace\SourceStackTrace.cpp @ 8]
14 00000074`fe5ffb60 00007ff7`bbd5269b SourceStackTrace!std::_Invoker_functor::_Call<void (__cdecl*)(void)>+0×15 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\type_traits @ 1571]
15 00000074`fe5ffb90 00007ff7`bbd525d4 SourceStackTrace!std::invoke<void (__cdecl*)(void)>+0×1b [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\type_traits @ 1571]
16 00000074`fe5ffbc0 00007ff7`bbd59784 SourceStackTrace!std::thread::_Invoke<std::tuple<void (__cdecl*)(void)>,0>+0×64 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.23.28105\include\thread @ 40]
17 00000074`fe5ffc10 00007ff9`74a17bd4 SourceStackTrace!thread_start<unsigned int (__cdecl*)(void *),1>+0×50 [minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 97]
18 00000074`fe5ffc40 00007ff9`752cced1 kernel32!BaseThreadInitThunk+0×14
19 00000074`fe5ffc70 00000000`00000000 ntdll!RtlUserThreadStart+0×21

Such Source Stack Trace frames has a similar structure as native module!function+offset format and can be mined as well including most of traditional stack trace patterns:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 181)

Wednesday, November 6th, 2019

Message Flow trace and log analysis pattern generalizes NetFlow to software narratives. We count messages based on the set of Adjoint Threads of Activity, for example PID.TID. This also subsumes network traces aggregated by Src.Dst. Individual single attributes can also be used, for example, aggregation by Thread of Activity, and also by Message Sets.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -