Archive for the ‘C++11’ Category

Crash Dump Analysis Patterns (Part 261)

Sunday, October 13th, 2019

Raw stack memory region analysis is more productive with Region Clusters analysis pattern. Here we apply simple clustering techniques to organize various region values into disjoint sets with chosen semantics. For our purposes simple sort suffices to generate such clusters that can be visually inspected. We take the same stack.csv file from Region Profile analysis pattern. It’s values are sorted and the results are shown in sorted order with corresponding count of occurrences and symbolic references (we use the internal version of Narrascope written in C++, a narrative debugger, but you can use your favorite scripting language here):

0 count: 13718
1 count: 273
2 count: 23
3 count: 22
4 count: 28
5 count: 9
6 count: 5
7 count: 18
8 count: 35
9 count: 5
a count: 24
b count: 12
c count: 4
d count: 3
e count: 1
f count: 28
10 count: 14
...
c0000034 count: 2
c0000388 count: 2
c01c0001 count: 1
c0a70000 count: 1
d0908070 count: 1
dcae0fa0 count: 1
e30000e3 count: 1
f80004fc count: 2
ffff5815 count: 2
fffffed3 count: 2
fffffffd count: 2
ffffffff count: 18
100000000 count: 6
100000001 count: 4
100001f80 count: 1
100001fa0 count: 16
100001fa4 count: 2
100003033 count: 2
100010000 count: 1
...
7ff700000000 count: 1
7ff700000001 count: 2
7ff70000000d count: 1
7ff747390000 Photoshop_exe count: 1
7ff74ebd4ec0 Photoshop_exe+0x7844ec0 count: 1
7ff74ef351c7 Photoshop_exe+0x7ba51c7 count: 1
7ff74ef4e2f0 Photoshop_exe+0x7bbe2f0 count: 1
7ff74ef4e5a9 Photoshop_exe+0x7bbe5a9 count: 1
...
7fff00000000 count: 21
7fff00000001 count: 7
7fff00000002 count: 1
7fff00000003 count: 1
7fff00000004 count: 1
7fff00000011 count: 1
7fff00000020 count: 1
7fff00000040 count: 3
7fff00000102 count: 1
7fff0000029e count: 3
7fff00140000 count: 1
7fff02000002 count: 1
7fff4782c33b libcef!GetHandleVerifier+0x61d7b count: 1
7fff4782c884 libcef!GetHandleVerifier+0x622c4 count: 1
7fff493749cc libcef!cef_time_to_timet+0x1a9228 count: 2
...
7fff9a0c1e57 GdiPlus!GpGraphics::MeasureString+0x333 count: 1
7fff9a128c2a GdiPlus!FastTextImager::MeasureString+0x32 count: 1
7fff9a174e18 GdiPlus!GpFontFamily::vftable' count: 2
7fff9b6055b3 DWrite!FontFace::GetDesignGlyphAdvances+0x57 count: 1
7fffa7e6c260 comctl32!ListBox_WndProc count: 5
7fffa7e6c357 comctl32!ListBox_WndProc+0xf7 count: 2
7fffb1373c18 npmproxy!INotifyNetworkListManagerEventsProxyVtbl+0x1b8 count: 1
7fffb2c14e96 msvcp140!_Mbrtowc+0x66 [f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c @ 156] count: 1
...
7fffc09f0359 ntdll!qsort+0x379 count: 1
7fffc09fa1e4 ntdll!woutput_s+0x8e8 count: 1
7fffc09fa297 ntdll!write_string+0x3f count: 1
7fffc09fbd30 ntdll!NtdllDefWindowProc_W count: 2
7fffc09fbf10 ntdll!NtdllDispatchHook_W count: 2
7fffc09ffc54 ntdll!KiUserCallForwarder+0x24 count: 1
7fffc09ffdb4 ntdll!KiUserCallbackDispatcherContinue count: 2
800000000000 count: 1
800000000001 count: 2
800063640000 count: 36
800066660000 count: 38
80006f6f0000 count: 2
800072720000 count: 8
800075750000 count: 1
974b00000000 count: 1
974b8118d10d count: 1
a76b00000000 count: 1
a76bb8365307 count: 1
a76bb8378c47 count: 1
a76bb8378f77 count: 1
a76bb837bfd7 count: 1
a8c300000000 count: 1
a8c311cf265f count: 1
...
30000000000000 count: 1
30000000310030 count: 1
30000300470048 count: 1
30002000100000 count: 1
3000300030007b count: 1
3000300031002d count: 1
30003000310031 count: 2
300031002d0037 count: 1
30003800390032 count: 3
31000000000000 count: 1
310000007d0036 count: 1
31002d00310037 count: 1
310032002d0035 count: 1
...
7fdf7fbd7f9c7f7b count: 2
8000800000000001 count: 1
8000800000001fa0 count: 1
8000800080000000 count: 6
8000800080008000 count: 52
80121a254b25250a count: 1
923800003f000000 count: 2
bf000000bf000000 count: 1
bff0000000000000 count: 2
e5b2a56118358cbe count: 2
ffff0072656c6c6f count: 1
fffffdb773438b57 count: 3
ffffff0000000005 count: 1
ffffff7bc010786f count: 1
ffffff7bc010787f count: 1
fffffffb00000000 count: 1
ffffffff00000000 count: 4
ffffffff00000001 count: 3
ffffffff00000005 count: 1
ffffffff00001fa0 count: 2
ffffffff4c494146 count: 2
ffffffffffffc3ce count: 1
fffffffffffffef6 count: 1
ffffffffffffff00 count: 2
ffffffffffffff01 count: 2
fffffffffffffffe count: 166
ffffffffffffffff count: 38

We can easily identify error values, module boundaries, and Regular Data. The sorting can also be done for double word or word values, for example to isolate errors or wide character values, but this will have to be seen whether it is useful.

This clustering approach can be depicted in the following idealized diagram:

The full output can be found here: stack-clusters.txt for stack.csv file.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 42n)

Friday, September 23rd, 2016

This is C++ condition variable Wait Chain pattern variant. When we have a waiting Blocked Thread stack trace that shows conditional variable implementation functions we are interested in the owner thread:

0:012> kL
# ChildEBP RetAddr
00 03a6f684 748d8ee9 ntdll!NtWaitForSingleObject+0xc
01 03a6f6f8 5f5fcba5 KERNELBASE!WaitForSingleObjectEx+0x99
02 03a6f70c 5f5fb506 msvcr120!Concurrency::details::ExternalContextBase::Block+0×37
03 03a6f778 639cea79 msvcr120!Concurrency::details::_Condition_variable::wait+0xab
04 03a6f7ac 639ceb58 msvcp120!do_wait+0×42
05 03a6f7c0 5c8c5a43 msvcp120!_Cnd_wait+0×10

WARNING: Stack unwind information not available. Following frames may be wrong.
06 03a6f7d0 5c8c4ee6 AppA!foo+0×48883
07 03a6f804 5c8c4bde AppA!foo+0×47d26
08 03a6f834 5c8c4b9c AppA!foo+0×47a1e
09 03a6f848 5c8c4a27 AppA!foo+0×479dc
0a 03a6f854 00dcc4e9 AppA!foo+0×47867
0b 03a6f86c 75823744 AppA+0×1c4e9
0c 03a6f880 76ef9e54 kernel32!BaseThreadInitThunk+0×24
0d 03a6f8c8 76ef9e1f ntdll!__RtlUserThreadStart+0×2f
0e 03a6f8d8 00000000 ntdll!_RtlUserThreadStart+0×1b

We see the thread is waiting for an event 4ac:

0:012> kv 2
# ChildEBP RetAddr Args to Child
00 03a6f684 748d8ee9 000004ac 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
01 03a6f6f8 5f5fcba5 000004ac ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0×99 (FPO: [SEH])

0:012> !handle 4ac
Handle 000004ac
Type Event

Instead of digging into implementation internals we show a different approach. We can use Constant Subtrace analysis pattern to find out possible owner thread candidates, and we can also check raw stack region Execution Residue of different threads for Place Trace of synchronization primitives and associated symbolic references (beware of Coincidental Symbolic Information though), and, if possible, Past Stack Traces involving synchronization.

If we look at Stack Trace Collection we can match the following thread that has the same Constant Subtrace as our original waiting thread above:

14 Id: 17a0.39d0 Suspend: 1 Teb: fee4f000 Unfrozen
# ChildEBP RetAddr
00 0679f9f0 748d9edc ntdll!NtReadFile+0xc
01 0679fa54 5c8f38f2 KERNELBASE!ReadFile+0xec
WARNING: Stack unwind information not available. Following frames may be wrong.
02 0679fa84 5c8f3853 AppA!foo+0x76732
03 0679fac8 5c8f37cd AppA!foo+0x76693
04 0679fae0 5c8c4a27 AppA!foo+0x7660d
05 0679faec 00dcc4e9 AppA!foo+0×47867
06 0679fb04 75823744 AppA+0×1c4e9
07 0679fb18 76ef9e54 kernel32!BaseThreadInitThunk+0×24
08 0679fb60 76ef9e1f ntdll!__RtlUserThreadStart+0×2f
09 0679fb70 00000000 ntdll!_RtlUserThreadStart+0×1b

When we dump raw stack data from all threads using this WinDbg script and search for 000004ac we find its occurrences in the raw stack that corresponds to thread #14 we already found:


[...]
TEB at fee4f000
ExceptionList: 0679fa44
StackBase: 067a0000
StackLimit: 0679e000

[…]
0679f90c 0679f918
0679f910 5f5a4894 msvcr120!Concurrency::details::SchedulerBase::CurrentContext+0×1e
0679f914 00000033
0679f918 0679f950
0679f91c 5f5a48ca msvcr120!Concurrency::details::LockQueueNode::LockQueueNode+0×2a
0679f920 0679f998
0679f924 071cf9ac
0679f928 0679f94c
0679f92c 5f5ff57e msvcr120!Concurrency::critical_section::_Acquire_lock+0×2e
0679f930 071cf9ac
0679f934 0679f994
0679f938 0679f998
0679f93c 00000001
0679f940 070f4bfc
0679f944 0679f970
0679f948 5f5ff41f msvcr120!Concurrency::critical_section::lock+0×31
0679f94c 0679f980
0679f950 5f5ff6dc msvcr120!Concurrency::critical_section::scoped_lock::scoped_lock+0×3b
0679f954 0679f998
0679f958 76f08bcc ntdll!NtSetEvent+0xc
0679f95c 748e30b0 KERNELBASE!SetEvent+0×10
0679f960 000004ac
0679f964 00000000
0679f968 0679f984
0679f96c 5f5fcbe6 msvcr120!Concurrency::details::ExternalContextBase::Unblock+0×3f
0679f970 000004ac
0679f974 03a6f750
0679f978 03a6f750
0679f97c 0679f9b4
0679f980 5f5fb70d msvcr120!Concurrency::details::_Condition_variable::notify_all+0×3f
0679f984 0679f9b4
0679f988 5f5fb722 msvcr120!Concurrency::details::_Condition_variable::notify_all+0×54
0679f98c 07481380
0679f990 05ba6f60
0679f994 071cf9ac
[…]

Both methods point to the same possible owner thread which is also blocked in reading a file.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -