Software Diagnostics Library

The Latent Structure pattern addresses situations where a memory region appears raw and untyped yet shows early, incomplete signs of structural organization. Signals, such as small or pointer-like values, alignment regularities, recurring byte sequences, partial strings, or fragments that resemble fields, suggest that a real structure might exist, but cannot yet be interpreted safely or confidently. Latent Structure represents the pre-suspect stage in structural diagnostics: the point where the analyst notices potential form but must resist premature interpretation. Acting too early risks misclassifying problems and misidentifying root causes. Several forces complicate this stage: partial overwrites, coincidental alignments, ABI or version mismatches, and cognitive biases that encourage overinterpretation. This analysis pattern, therefore, emphasizes careful, hypothesis-driven exploration using techniques such as tentative structure casting, pointer-chain heuristics, checks for internal semantic coherence, software internals, and domain knowledge, all without assuming the structure’s validity. When enough evidence accumulates, a Latent Structure transitions into a Suspect Structure (subject of the next analysis pattern), where it becomes testable.

For example, we may see these fragment in Execution Residue:

00000029`e7efeb00 00001f80`0010004b
00000029`e7efeb08 0053002b`002b0033
00000029`e7efeb10 00000242`002b002b

Finally, we write the formal pattern structure card.

Intent

Detect hidden or unclear structural organization in raw memory regions that exhibit early indicators of structure-like form but whose types are not yet known.

Context

Appears in:
Execution Residue, Pointer Cone, Memory Region, and Region Strata.

Problem

A memory dump shows a region of raw bytes without explicit type information that contains hints suggesting that a structure may be present. Prematurely interpreting such memory can lead to false positives, misclassification, incorrect casting, and a chain of misleading hypotheses.

Forces

Data:

• Memory may contain partial structures
• Overwrites blur structure boundaries
• Random-looking regions may hide structured subregions

Semantics:

• Pointer-like values may be real or coincidental
• Partial strings
• Field alignments may appear regular due to chance

Modules:

• Coincidental symbols
• ABI or version mismatches

Cognitive biases:

• Insufficient domain knowledge
• Premature suspicion

Symptoms

• Structural hints in bytes
• Pointer-like values
• Strings and identity hints
• Alignment and regularity
• Recurring patterns across multiple memory locations
• Partial structure validity
• Incomplete or corrupt-like structure

Resolution Strategies

• Structure casting
• Heuristic field and pointer chain analysis
• Verification of internal semantic coherence

Resulting Context

Structure becomes Suspect and testable for validity.

Related Patterns

Hidden Artifact Patterns, Corrupt Structure, Module Hint, Falsity and Coincidence Patterns, Shared Buffer Overwrite, Value References, Small Value, Design Value, Shared Structure, and Regular Data.

Formal Card

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we get memory dumps, we are interested in Dump Context: an issue description, how its memory dump was collected, overall system information, related Paratext, and previous memory dumps and their analyses. This contextual information can be organized as a checklist to ensure situational awareness, diagnostic quality, and reduce the number of information request roundtrips.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of collection-related analysis patterns is necessary, so I created this post:

• Stack Trace Collection (unmanaged space)
• Stack Trace Collection (managed space)
• Stack Trace Collection (predicate)
• Stack Trace Collection (I/O requests)
• Stack Trace Collection (CPUs)
• Last Error Collection
• Module Collection
• Module Collection (predicate)
• Structure Field Collection
• Rough Stack Trace Collection (unmanaged space)
• Exception Collection
• Interrupt Stack Collection
• DPC Stack Collection

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When analyzing Execution Residue we are interested in valid symbolic references, for example, valid return addresses for Rough Stack Trace and Past Stack Trace reconstruction. Some of them may be Coincidental Symbolic Information that we check with backward disassembly. However, some symbolic references may be Function Pointers if forward disassembly starts with valid function prologue. Some references may also point to the start of exception processing in the middle of a normal function body. These two variants are illustrated with the following two addresses taken from the raw stack data when debugging an x64 Linux process in WSL using the latest version of WinDbg:

0:000> u 00007f73`e45820b0
libgcc_s_so!Unwind_Backtrace+0x7c0:
00007f73`e45820b0 push r15
00007f73`e45820b2 push r14
00007f73`e45820b4 push r13
00007f73`e45820b6 push r12
00007f73`e45820b8 push rbp
00007f73`e45820b9 push rbx
00007f73`e45820ba sub rsp,58h
00007f73`e45820be mov ecx,dword ptr [rdx+28h]

0:000> u 00005564`99d582c8
ud5bv3!start_modeling+0xd7:
00005564`99d582c8 mov rdi,rax
00005564`99d582cb call ud5bv3!_cxa_begin_catch$plt (00005564`99d58040)
00005564`99d582d0 call ud5bv3!_cxa_end_catch$plt (00005564`99d580b0)
00005564`99d582d5 jmp ud5bv3!start_modeling+0x64 (00005564`99d58255)
00005564`99d582da leave
00005564`99d582db ret
ud5bv3!new_feature:
00005564`99d582dc push rbp
00005564`99d582dd mov rbp,rsp

0:000> ub 00005564`99d582c8
ud5bv3!start_modeling+0xb5:
00005564`99d582a6 mov rdi,rax
00005564`99d582a9 call ud5bv3!sem_close$plt (00005564`99d580a0)
00005564`99d582ae mov rax,qword ptr [rbp-8]
00005564`99d582b2 mov rdi,rax
00005564`99d582b5 call ud5bv3!sem_close$plt (00005564`99d580a0)
00005564`99d582ba lea rdi,[ud5bv3!IO_stdin_used+0x4 (00005564`99d59004)]
00005564`99d582c1 call ud5bv3!sem_unlink$plt (00005564`99d580c0)
00005564`99d582c6 jmp ud5bv3!start_modeling+0xe9 (00005564`99d582da)

As we see, some Function Pointers may have symbolic name plus some offset if they don’t have associated symbols.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This analysis pattern is a Linux variant of the previous Not My Version Windows crash dump analysis pattern. In case of segmentation faults/core dumps with Stack Traces involving OS and 3rd-party shared libraries it is worth noting directories they come from:

(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007fad170d5950 0x00007fad170e2dc8 Yes (*) /lib64/lib3rdparty.so.0
0x00007fad170c3130 0x00007fad170c3eb5 Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007fad16f24320 0x00007fad1706a14b Yes /lib/x86_64-linux-gnu/libc.so.6
0x00007fad17103090 0x00007fad17120b50 Yes /lib64/ld-linux-x86-64.so.2
0x00007fad16bfd300 0x00007fad16c03578 Yes /lib/x86_64-linux-gnu/libnss_files.so.2

If you use WinDbg to analyze Linux core dumps, you can use lmv command or its variant. It is worth noting that referenced libraries can be symbolic links to different versions than you expect. Unfortunately, it is not so easy to extract library timestamps, so it is recommended to additionally use Paratext analysis pattern to supply such information.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If we dump contents of memory regions such as thread stacks we get different region memory addresses due to ASLR. To compare such regions from different executions we need to make them Normalized Regions replacing varying address parts with a common symbolic denominator, for example, 0-based addresses (or some other non-digital symbols to account of the possibility of already existing similar 0-based addresses) and adjusting address offsets appropriately. For example, the following raw stack region from Foreign Stack analysis pattern (we do not normalize foreign stack addresses below):

00000055`adbfe000 00000000`00000000
00000055`adbfe008 00000000`00000000
00000055`adbfe010 00000000`00000000
00000055`adbfe018 00000000`00000000
00000055`adbfe020 00000000`00000000
…
00000055`adbffa58 00000000`00000008
00000055`adbffa60 00000055`adbffb28
00000055`adbffa68 00007ff7`96e116a3 module!func+0×13
00000055`adbffa70 00000055`ad8ffde0
00000055`adbffa78 00000000`00000000
00000055`adbffa80 00000000`00000000
00000055`adbffa88 00007ffa`28ecc9a8 ucrtbase!`string’
00000055`adbffa90 00000000`00000000
00000055`adbffa98 00007ff7`96e116c3 module!thread_proc+0×13
00000055`adbffaa0 00000055`ad8ffde0
00000055`adbffaa8 00000055`adbffb28
00000055`adbffab0 ffffffff`ffffffff
00000055`adbffab8 00007ffa`28fa0748 KERNELBASE!GetAppModelPolicy+0×18
00000055`adbffac0 00000055`adbffba0
00000055`adbffac8 00007ff7`96e12657 module!std::invoke<void (__cdecl*)(int *),int *>+0×27
00000055`adbffad0 00000055`ad8ffde0
00000055`adbffad8 00000055`adbffb00
00000055`adbffae0 00000055`adbffb18
…

--------`00000000 00000000`00000000
--------`00000008 00000000`00000000
--------`00000010 00000000`00000000
--------`00000018 00000000`00000000
--------`00000020 00000000`00000000
…
--------`00001a58 00000000`00000008
--------`00001a60 --------`00001b28
--------`00001a68 00007ff7`96e116a3 module!func+0×13
--------`00001a70 00000055`ad8ffde0
--------`00001a78 00000000`00000000
--------`00001a80 00000000`00000000
--------`00001a88 00007ffa`28ecc9a8 ucrtbase!`string’
--------`00001a90 00000000`00000000
--------`00001a98 00007ff7`96e116c3 module!thread_proc+0×13
--------`00001aa0 00000055`ad8ffde0
--------`00001aa8 --------`00001b28
--------`00001ab0 ffffffff`ffffffff
--------`00001ab8 00007ffa`28fa0748 KERNELBASE!GetAppModelPolicy+0×18
--------`00001ac0 --------`00001ba0
--------`00001ac8 00007ff7`96e12657 module!std::invoke<void (__cdecl*)(int *),int *>+0×27
--------`00001ad0 00000055`ad8ffde0
--------`00001ad8 --------`00001b00
--------`00001ae0 --------`00001b18
…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Values in memory regions, when interpreted as addresses, such as stack can be classified according to their categories such as zero, module (code and data), heap, stack, Foreign Stack, Small Value, error, and out-of-range (for example, kernel space address in user space). Such classification can be encoded and optionally visualized as Region Spectrum:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The content of crash dump analysis patterns is available in edited and revised PDF format:

https://www.patterndiagnostics.com/encyclopedia-crash-dump-analysis-patterns

Raw stack memory region analysis is more productive with Region Clusters analysis pattern. Here we apply simple clustering techniques to organize various region values into disjoint sets with chosen semantics. For our purposes simple sort suffices to generate such clusters that can be visually inspected. We take the same stack.csv file from Region Profile analysis pattern. It’s values are sorted and the results are shown in sorted order with corresponding count of occurrences and symbolic references (we use the internal version of Narrascope written in C++, a narrative debugger, but you can use your favorite scripting language here):

0 count: 13718
1 count: 273
2 count: 23
3 count: 22
4 count: 28
5 count: 9
6 count: 5
7 count: 18
8 count: 35
9 count: 5
a count: 24
b count: 12
c count: 4
d count: 3
e count: 1
f count: 28
10 count: 14
...
c0000034 count: 2
c0000388 count: 2
c01c0001 count: 1
c0a70000 count: 1
d0908070 count: 1
dcae0fa0 count: 1
e30000e3 count: 1
f80004fc count: 2
ffff5815 count: 2
fffffed3 count: 2
fffffffd count: 2
ffffffff count: 18
100000000 count: 6
100000001 count: 4
100001f80 count: 1
100001fa0 count: 16
100001fa4 count: 2
100003033 count: 2
100010000 count: 1
...
7ff700000000 count: 1
7ff700000001 count: 2
7ff70000000d count: 1
7ff747390000 Photoshop_exe count: 1
7ff74ebd4ec0 Photoshop_exe+0x7844ec0 count: 1
7ff74ef351c7 Photoshop_exe+0x7ba51c7 count: 1
7ff74ef4e2f0 Photoshop_exe+0x7bbe2f0 count: 1
7ff74ef4e5a9 Photoshop_exe+0x7bbe5a9 count: 1
...
7fff00000000 count: 21
7fff00000001 count: 7
7fff00000002 count: 1
7fff00000003 count: 1
7fff00000004 count: 1
7fff00000011 count: 1
7fff00000020 count: 1
7fff00000040 count: 3
7fff00000102 count: 1
7fff0000029e count: 3
7fff00140000 count: 1
7fff02000002 count: 1
7fff4782c33b libcef!GetHandleVerifier+0x61d7b count: 1
7fff4782c884 libcef!GetHandleVerifier+0x622c4 count: 1
7fff493749cc libcef!cef_time_to_timet+0x1a9228 count: 2
...
7fff9a0c1e57 GdiPlus!GpGraphics::MeasureString+0x333 count: 1
7fff9a128c2a GdiPlus!FastTextImager::MeasureString+0x32 count: 1
7fff9a174e18 GdiPlus!GpFontFamily::vftable' count: 2
7fff9b6055b3 DWrite!FontFace::GetDesignGlyphAdvances+0x57 count: 1
7fffa7e6c260 comctl32!ListBox_WndProc count: 5
7fffa7e6c357 comctl32!ListBox_WndProc+0xf7 count: 2
7fffb1373c18 npmproxy!INotifyNetworkListManagerEventsProxyVtbl+0x1b8 count: 1
7fffb2c14e96 msvcp140!_Mbrtowc+0x66 [f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c @ 156] count: 1
...
7fffc09f0359 ntdll!qsort+0x379 count: 1
7fffc09fa1e4 ntdll!woutput_s+0x8e8 count: 1
7fffc09fa297 ntdll!write_string+0x3f count: 1
7fffc09fbd30 ntdll!NtdllDefWindowProc_W count: 2
7fffc09fbf10 ntdll!NtdllDispatchHook_W count: 2
7fffc09ffc54 ntdll!KiUserCallForwarder+0x24 count: 1
7fffc09ffdb4 ntdll!KiUserCallbackDispatcherContinue count: 2
800000000000 count: 1
800000000001 count: 2
800063640000 count: 36
800066660000 count: 38
80006f6f0000 count: 2
800072720000 count: 8
800075750000 count: 1
974b00000000 count: 1
974b8118d10d count: 1
a76b00000000 count: 1
a76bb8365307 count: 1
a76bb8378c47 count: 1
a76bb8378f77 count: 1
a76bb837bfd7 count: 1
a8c300000000 count: 1
a8c311cf265f count: 1
...
30000000000000 count: 1
30000000310030 count: 1
30000300470048 count: 1
30002000100000 count: 1
3000300030007b count: 1
3000300031002d count: 1
30003000310031 count: 2
300031002d0037 count: 1
30003800390032 count: 3
31000000000000 count: 1
310000007d0036 count: 1
31002d00310037 count: 1
310032002d0035 count: 1
...
7fdf7fbd7f9c7f7b count: 2
8000800000000001 count: 1
8000800000001fa0 count: 1
8000800080000000 count: 6
8000800080008000 count: 52
80121a254b25250a count: 1
923800003f000000 count: 2
bf000000bf000000 count: 1
bff0000000000000 count: 2
e5b2a56118358cbe count: 2
ffff0072656c6c6f count: 1
fffffdb773438b57 count: 3
ffffff0000000005 count: 1
ffffff7bc010786f count: 1
ffffff7bc010787f count: 1
fffffffb00000000 count: 1
ffffffff00000000 count: 4
ffffffff00000001 count: 3
ffffffff00000005 count: 1
ffffffff00001fa0 count: 2
ffffffff4c494146 count: 2
ffffffffffffc3ce count: 1
fffffffffffffef6 count: 1
ffffffffffffff00 count: 2
ffffffffffffff01 count: 2
fffffffffffffffe count: 166
ffffffffffffffff count: 38

We can easily identify error values, module boundaries, and Regular Data. The sorting can also be done for double word or word values, for example to isolate errors or wide character values, but this will have to be seen whether it is useful.

This clustering approach can be depicted in the following idealized diagram:

The full output can be found here: stack-clusters.txt for stack.csv file.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Manual analysis of Execution Residue in stack regions can be quite daunting so some basic statistics on the distribution of address, value, and symbolic information can be useful. To automate this process we use Pandas python library and interpret preprocessed WinDbg output of dps and dpp commands as DataFrame object:

import pandas as pd
import pandas_profiling

df = pd.read_csv("stack.csv")
html_file = open("stack.html", "w")
html_file.write (pandas_profiling.ProfileReport(df).to_html())
html_file.close()

We get a rudimentary profile report: stack.html for stack.csv file. The same was also done for Address, Address/Value, Value, Symbol output of dpp command: stack4columns.html for stack4columns.csv file.

We call this analysis pattern Region Profile since any Memory Region can be used. This analysis pattern is not limited to Python/Pandas and different libraries/scripts/script languages can also be used for statistical and correlational profiling. We plan to provide more examples here in the future.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Some Stack Traces reveal a functional purpose, for example, painting, querying a database, doing HTML or JavaScript processing, loading a file, printing, or something else. Such traces from various Stack Trace Collections (unmanaged, managed, predicate, CPUs) may be compared for similarity and help with analysis patterns, such as examples in Wait Chain (C++11 condition variable, SRW lock), finding semantically Coupled Processes, and many others where we look at the meaning of stack trace frame sequences to relate them to each other. We call this analysis pattern Stack Trace Motif by analogy with Motif trace and log analysis pattern. Longer stack traces may contain several Motives and also Technology-Specific Subtraces (for example, COM interface invocation frames).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Often, for Inter-Correlational trace and log analysis, we need to make sure that we have synchronized traces. The one version of Unsynchronized Traces analysis pattern is depicted in the following diagram where one trace ends  (possibly Truncated Trace) before the start of another trace and both were traced within one hour:

If tracing was done in different time zones with different local times specified in logs we can determine whether the traces are synchronized (when time zone information is not available in Basic Facts) by looking at minutes as shown for the third trace in the diagram above. This technique can also be used in trace calibration (see Calibrating Trace).

There is a similar analysis pattern for memory analysis called Unsynchronized Dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Adjoint Space and Trace Fibration analysis patterns may be useful in cases where complementing traces with memory dumps and vice versa provides better insight into software behavior. Ideally, every trace statement should have Adjoint Space but this is not feasible practically. The solution is to save memory regions surrounding trace message data, for example, structures referenced by pointers. This can be done either for each message or for selected Message Sets. Such memory data can be embedded inside logs as State Dump, Trace Extension or Inter-Correlation with a binary log of such memory fragments. This looks like a mapping between trace messages and memory objects. We call this analysis pattern Trace Presheaf by analogy with presheaves in mathematics. In case Adjoint Spaces are available (for example, memory dumps) such memory buffers can be written to memory by a debugger (in Windows by .readmem WinDbg command) and examined in association with the rest of Adjoint Space. This is illustrated in the following diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Recently we found a correlation between software trace with high Statement Density and Current of Periodic Error with uniform Error Distribution and process heap Memory Leak suspected from  memory dump analysis. If we metaphorically view periodic errors as “frequency” and the size of a heap as “mass” we may see that the growth of “frequency” correlates with the growth of “mass” and vice versa. Since frequency is inversely proportional to wave length we see a metaphorical analog to Louis de Broglie’s wave-particle duality. In general, as we already pointed in the discussion of narrativity and spatiality of software execution artifacts (see also Software Trace and Memory Dump Analysis seminar), software traces / logs and memory dumps can be seen as “dual” to each other according (metaphorically again) to de Broglie’s “duality of the laws of nature”. So we name this analysis pattern De Broglie Trace Duality since some memory dump regions can be considered of a general trace nature. Our correlation can be depicted in this diagram:

Practically, when we see Memory Consumption Patterns (but don’t know their source / root cause yet) we can ask for traces and logs, and in the case of frequent Periodic Errors found there we can suggest troubleshooting steps that may serve as a resolution or workaround.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a Linux variant of Module Stack Trace pattern previously described for Windows platform. Linux core dumps are Abridged Dumps by default with shared libraries code and paged out pages missing. To enable saving full process dumps use this command (see core man page for more details):

[training@localhost CentOS]$ echo 0x7f > /proc/$$/coredump_filter

Compare the file sizes for sleep process core dump generated before (core.3252) and after (core.3268) changing coredump_filter value:

[training@localhost CentOS]$ ls -l
-rwxrwxrwx. 1 root root 323584 Oct 3 07:39 core.3252
-rwxrwxrwx. 1 root root 103337984 Oct 3 07:40 core.3268

Although GDB is not able to get symbolic stack trace for both dumps above due to the absence of symbols, CODA tool is able to show stack trace variant with modules (with Reduced Symbolic Information):

(gdb) bt
#0 0x00000032bd4accc0 in ?? ()
#1 0x0000000000403ce8 in ?? ()
#2 0x00000000000004d2 in ?? ()
#3 0x0000000000000000 in ?? ()

[training@localhost CentOS]$ ./coda/coda -i core.3268
Welcome to coda interactive command line.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND.
Supported on x86_64-linux.
Coredump generated by command line => sleep 1234
coda > bt
[0] 0x00000032bd4accc0 <0x00000032bd4accb0 - 0x00000032bd4acd0e> __nanosleep+0x10 [RO TEXT]:/lib64/libc.so.6
[1] 0×0000000000403ce8 <0×0000000000403c40 - 0×0000000000403cf3> close_stdout+0×2378 [RO TEXT]:sleep
[2] 0×000000000040336d <0×00000000004032c0 - 0×00000000004033e9> close_stdout+0×19fd [RO TEXT]:sleep
[3] 0×00000000004016bc <0×00000000004014c0 - 0×0000000000401775> usage+0×3fc [RO TEXT]:sleep
[4] 0×00000032bd41ed1d <0×00000032bd41ec20 - 0×00000032bd41ede7> __libc_start_main+0xfd [RO TEXT]:/lib64/libc.so.6
[5] 0×00000000004011f9 <0×00000000004012c0 - 0×00000000004014bb> ?? [RO TEXT]:sleep
[6] 0×00007fff89d82a68 <————-RANGE UNKNOWN————-> ?? [RW DATA]:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Long time go we introduced the notion of Collapsed Stack Trace (Volume 3, page 381) when all functions are removed from Stack Trace (for example, kc WinDbg command) and remaining repeated modules are removed similar to Quotient Trace analysis pattern. It is similar to Stack Trace Signature with frame count set to 1. We originally planned to call this pattern Compact Stack (Trace) and it was on our list of possible future analysis patterns. This came to our attention again while preparing “Theoretical Software Diagnostics” book and we decided to publish it under the name Quotient Stack Trace as a specialization of the more general trace and log analysis pattern.

Such a pattern may be useful for the analysis of module Wait Chains.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Stack traces resemble functions: they have prolog, body, and epilog. Frame trace is also similar to trace Partition analysis pattern. Bottom stack subtrace plays the role of prolog, for example, thread initialization and RPC call stub dispatch. Middle stack subtrace plays the role of body or core, for example, application specific function calls invoked by RPC. Top stack subtrace plays the role of epilogue, for example, system calls. Such stack trace partition is useful for stack trace matching, especially when symbols are not available. In such a case Stack Trace Signature of module names and their frame counts may help (together with Crash Signature where appropriate):

The following stack trace may be split into TMB (pronounced Tomb):

0:001> kc
# Call Site
00 ntdll!RtlEnterCriticalSection
01 ModuleA
02 ModuleA
03 ModuleA
04 ModuleA
05 ModuleA
06 ModuleA
07 ModuleA
08 ModuleA
09 ModuleA
0a ModuleA
0b ModuleA
0c ModuleA
0d ModuleA
0e ModuleA
0f ModuleA
10 rpcrt4!Invoke
11 rpcrt4!NdrStubCall2
12 rpcrt4!NdrServerCall2
13 rpcrt4!DispatchToStubInCNoAvrf
14 rpcrt4!RPC_INTERFACE::DispatchToStubWorker
15 rpcrt4!RPC_INTERFACE::DispatchToStub
16 rpcrt4!RPC_INTERFACE::DispatchToStubWithObject
17 rpcrt4!LRPC_SCALL::DispatchRequest
18 rpcrt4!LRPC_SCALL::HandleRequest
19 rpcrt4!LRPC_SASSOCIATION::HandleRequest
1a rpcrt4!LRPC_ADDRESS::HandleRequest
1b rpcrt4!LRPC_ADDRESS::ProcessIO
1c rpcrt4!LrpcIoComplete
1d ntdll!TppAlpcpExecuteCallback
1e ntdll!TppWorkerThread
1f kernel32!BaseThreadInitThunk
20 ntdll!RtlUserThreadStart

It has the following signature:

T<ntdll,1>M<ModuleA,15>B<rpcrt4,13>B<ntdll,2>B<kernel32,1>B<ntdll,1>

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a Linux variant of Local Buffer Overflow pattern previously described for Mac OS X and Windows platforms. Most of the time simple mistakes in using memory and string manipulation functions are easily detected by runtime. The more sophisticated example which overwrites stack trace without being detected involves overwriting indirectly via a pointer to a local buffer passed to the called function. In such cases we might see incorrect and truncated stack traces:

(gdb) bt
#0  0×0000000000000000 in ?? ()
#1  0×0000000000000000 in ?? ()

(gdb) x/100a $rsp
[...]
0x7fc3dd9dece8: 0x0 0x0
0x7fc3dd9decf8: 0x0 0x0
0x7fc3dd9ded08: 0x0 0x0
0x7fc3dd9ded18: 0x0 0x0
0x7fc3dd9ded28: 0×7fc3dd9ded48 0×4005cc <procA+40>
0×7fc3dd9ded38: 0×422077654e20794d 0×7542207265676769
0×7fc3dd9ded48: 0×72656666 0×0
0×7fc3dd9ded58: 0×0 0×0
0×7fc3dd9ded68: 0×0 0×0
0×7fc3dd9ded78: 0×0 0×0
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a Linux variant of Stack Overflow (user mode) pattern previously described for Mac OS X and Windows platforms:

(gdb) bt
#0  0x00000000004004fb in procF ()
#1  0x000000000040054b in procF ()
#2  0x000000000040054b in procF ()
#3  0x000000000040054b in procF ()
#4  0x000000000040054b in procF ()
#5  0x000000000040054b in procF ()
#6  0x000000000040054b in procF ()
#7  0x000000000040054b in procF ()
#8  0x000000000040054b in procF ()
#9  0x000000000040054b in procF ()
#10 0x000000000040054b in procF ()
#11 0x000000000040054b in procF ()
#12 0x000000000040054b in procF ()
#13 0x000000000040054b in procF ()
#14 0x000000000040054b in procF ()
#15 0x000000000040054b in procF ()
#16 0x000000000040054b in procF ()
[...]

(gdb) bt -10
#15409 0x000000000040054b in procF ()
#15410 0x000000000040054b in procF ()
#15411 0x000000000040054b in procF ()
#15412 0x000000000040055b in procE ()
#15413 0x0000000000400575 in bar_one ()
#15414 0x0000000000400585 in foo_one ()
#15415 0x000000000040059d in thread_one ()
#15416 0x0000000000401690 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#15417 0x0000000000432549 in clone ()
#15418 0x0000000000000000 in ?? ()

In case of a stack overflow the stack pointer is decremented beyond the stack region boundary into an non-accessible region so any stack memory access triggers an access violation:

(gdb) x $rsp
0×7eff46109ec0: 0×0

(gdb) frame 1
#1  0x000000000040054b in procF ()

(gdb) x $rsp
0×7eff4610a0e0: 0×0

(gdb) maintenance info sections
[...]
Core file:
[...]
0×7eff46109000->0×7eff4610a000 at 0×02034000: load13 ALLOC LOAD READONLY HAS_CONTENTS
0×7eff4610a000->0×7eff4690a000 at 0×02035000: load14 ALLOC LOAD HAS_CONTENTS
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a Linux variant of Coincidental Symbolic Information pattern previously described for Mac OS X and Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:

(gdb) x/i 0x4005e6
0x4005e6 <_Z6work_3v+9>: pop    %rbp

(gdb) disassemble 0x4005e6
Dump of assembler code for function _Z6work_3v:
0x00000000004005dd <+0>: push   %rbp
0x00000000004005de <+1>: mov    %rsp,%rbp
0x00000000004005e1 <+4>: callq  0×4005d2 <_Z6work_4v>
0×00000000004005e6 <+9>: pop    %rbp
0×00000000004005e7 <+10>: retq
End of assembler dump.

(gdb) x/4i 0x49c740-4
0x49c73c: add    %al,(%rax)
0x49c73e: add    %al,(%rax)
0×49c740 <default_attr>: add    %al,(%rax)
0×49c742 <default_attr+2>: add    %al,(%rax)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -