Archive for April, 2014

Falsity and Coincidence Patterns

Monday, April 28th, 2014

A page to reference all different kinds of coincidence and falsity related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 205)

Saturday, April 26th, 2014

When calculating effective addresses such as [r10+10h] or [rax+rcx*12h+40h] to show their value in the output of some commands such as .trap or .cxr a debugger uses CPU register values from a saved trap frame or context structure. If such information is invalid the reported effective address doesn’t correspond to the real one during code execution. So we call this analysis pattern False Effective Address similar to False Function Parameters. Therefore, if a fault address is saved during bugcheck or exception processing it may not correspond to the output of some commands where such calculation is necessary. For example, in a bugcheck parameter we have this referenced memory address:

Arg1: fffffadda17d001d, memory referenced

but the output of .trap command shows a NULL pointer address:

NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
[...]
movzx eax,word ptr [rax+10h] 0010=????

Usually we are lucky and an effective address is correct despite the warning such as here and here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 80)

Monday, April 21st, 2014

Sometimes we know from Basic Facts some data or activity we seek to identify in different traces collected together to perform inter-correlational analysis. It can be a shared file name, a named synchronization object, a locked file with sharing violations, a common virtual address in kernel space, or just some activity notification. We call this pattern Shared Point by analogy with intersecting curves in some abstract space.

It is similar to Linked Messages pattern but is more high level and not confined to a common parameter (can be an action description).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 79)

Sunday, April 20th, 2014

We mostly analyse real messages in software traces and logs. In such message streams we may see easily detectable Discontinuity patterns. However, in some cases it is beneficial to analyse the absence of messages. Message stream is not uniform, there may be different currents. If time resolution is 1 ms, for example, then we may have a current N msg/ms or in the case of lesser current, such as, 0.5 msg/ms we have the so called Silent Messages (—-):

[...]
11 ms: message
12 ms: ----
13 ms: message
14 ms: ----
15 ms: message
16 ms: message
17 ms: ----
18 ms: ----
19 ms: message
[...]

So, by a silent message we understand the possible message that would occupy the minimal time resolution gap. If we look at the following illustration we would see that the whole pattern analysis apparatus can be applied to the analysis of distribution of silent messages.

Silent Messages pattern is different from Discontinuity pattern because the latter is about large unexpected silences and Sparse Trace which is about missing trace statements from source code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 190b)

Saturday, April 19th, 2014

While working on Thread Cluster pattern I realized that we need a predicate version of Module Collection pattern, similar to the predicate version of Stack Trace Collection pattern. A predicate can be anything: a company vendor, semantic proximity, functionality such as printing, remote file management, and so on. Such module sub-collections can be used instead of modules in more complex patterns: an example of software diagnostics pattern substitution and composition. For example, we might be able to identify a possible coupling between 2 semantically different module groups explained by IPC Wait Chains such as on this diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 204)

Friday, April 18th, 2014

One of useful patterns for the analysis of system hangs is Thread Waiting Time. If there are many such threads of interest they can be partitioned by waiting time and modules of interest from their stack traces. Modules of interest may include Directing, Coupled, Blocking, Top, and/or Problem modules depending on the problem description. We call the resulting composite pattern Thread Cluster. Extra-dimensional information can also be added such as the number of threads having the same or similar waiting time and other attributes by using different colours. For example, on this diagram, illustrating a real system hang, we see clustering of threads running through one 3rd-party module of interest and having the longest waiting time. Also we are able to identify possibly coupled (semantically related) threads running through another module of interest:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 42l)

Wednesday, April 9th, 2014

This is a variation of a general Wait Chain pattern related to CLR threads. When looking at Stack Trace Collection from a complete memory dump we may find threads using a monitor synchronization mechanism:

[... 32-bit ...]
09d2e908 6ba4d409 clr!CLREvent::WaitEx+0x106
09d2e91c 6bb90160 clr!CLREvent::Wait+0x19
09d2e9ac 6bb90256 clr!AwareLock::EnterEpilogHelper+0xa8
09d2e9ec 6bb9029b clr!AwareLock::EnterEpilog+0x42
09d2ea0c 6ba90f78 clr!AwareLock::Enter+0x5f
09d2eaa8 05952499 clr!JIT_MonEnterWorker_Portable+0xf8
[…]

or

[... 64-bit ...]
00000000`2094e230 000007fe`eedc3e3a clr!CLREvent::WaitEx+0xc1
00000000`2094e2d0 000007fe`eedc3d43 clr!AwareLock::EnterEpilogHelper+0xca
00000000`2094e3a0 000007fe`eee3e613 clr!AwareLock::EnterEpilog+0x63
00000000`2094e400 000007ff`007f4c38 clr!JIT_MonEnterWorker_Portable+0×14f
[…]

When seeing such threads we may ask for a process memory dump to perform .NET memory dump analysis using SOS or other WinDbg extensions such as in Deadlock pattern example for CLR 2 (mscorwks).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -