Archive for February, 2017

Trace Analysis Patterns (Part 140)

Tuesday, February 21st, 2017

Having chosen a trace message, we are interested in its Message Context which can span all “continuous” messages before and after from the same Thread of Activity. We it Activity Quantum which is variable and independent from the so-called CPU quantums. Different messages from Activity Quantum may be executed on different CPUs. The following diagram depicts this analysis pattern:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 248)

Sunday, February 19th, 2017

If OS is not inside a virtual machine it is difficult to get consistent live snapshots of physical memory (see Inconsistent Dump analysis pattern). Mirror dump options in LiveKd can save a consistent kernel memory dump. Then we can either use Fiber Bundle technique of saving individual process memory dumps or create inconsistent complete memory dump using LiveKd or both. We call this pattern Mirror Dump Set.

We can identify mirror dump with the following current stack trace:

0: kd> version
...
64-bit Kernel bitmap dump: ...
...

0: kd> k
# Child-SP RetAddr Call Site
00 ffffd000`26121700 fffff803`cf5f5ee3 nt!IopLiveDumpEndMirroringCallback+0x7f
01 ffffd000`26121750 fffff803`cf60561b nt!MmDuplicateMemory+0x807
02 ffffd000`26121830 fffff803`cf851c60 nt!IopLiveDumpCaptureMemoryPages+0x53
03 ffffd000`26121890 fffff803`cf447443 nt!IoCaptureLiveDump+0xf8
04 ffffd000`261218e0 fffff803`cf8ceb0d nt!DbgkCaptureLiveKernelDump+0x2e7
05 ffffd000`26121970 fffff803`cf3debb3 nt!NtSystemDebugControl+0x3f5
06 ffffd000`26121a90 00007ffa`2925205a nt!KiSystemServiceCopyEnd+0x13
07 000000a3`5bcddb48 00000000`00000000 0x00007ffa`2925205a

In one analysis case, we got such a set where we analyzed ALPC Wait Chains with user space stack traces in a complete memory having the endpoint blocked in a filter driver. But the search for stack traces having filter manager in their frames failed due to inconsistency:

0: kd> version
...
64-bit Full kernel dump: ...
...

0: kd> !stacks 2 FltMgr
...
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
...

So we found such kernel space stack traces from the consistent mirror dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 139)

Wednesday, February 15th, 2017

Here we introduce Delay Dynamics analysis pattern. It is not an oxymoron and dynamics is referred to by what actually happens during the delay (Discontinuity with Time Delta) in other Threads of Activity as depicted in the following diagram:

Instead of threads various Adjoint Threads of Activity may also be inspected.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 138)

Tuesday, February 7th, 2017

One of the trace attributes we didn’t pay much attention to in the past is CPU. This column is present in some ETW-based trace implementations such as Citrix CDF traces. As any trace attribute, it can be used to form Adjoint Thread of Activity (as all messages from code executed on that particular CPU). As we already considered threads as braids, we use braid groups as a further metaphor. In our case we combine CPUs and threads into one group which uses permutation for CPU scheduling. Instead of permutations, twists may be modeled as changes of threads. The Braid Group analysis pattern is illustrated in the following diagram:

This is a preliminary description of the analysis pattern. We plan to elaborate on it in further case studies. For example, instead of multithreading we can use multibraiding.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 247)

Sunday, February 5th, 2017

We got the idea of Unified Stack Trace analysis pattern from Flame Graphs. Like the latter, we combine Stack Trace Collection into one aggregate trace, but we may use the same length for repeated frames and may use different color intensities to present multiplicities. Different frame height may also be used to unify top frames such as waiting API. Different collections may be used in addition to database-like stack traces (Unmanaged, Managed, Predicate, I/O, CPU). The collections may be composed from different varieties of stack traces, such as General, Managed, Module, Quotient, Filters).

As a very simple example, consider this Stack Trace Collection from Notepad:

0:003> ~*kc

0  Id: 984.994 Suspend: 1 Teb: 00007ff6`f411d000 Unfrozen
# Call Site
00 USER32!NtUserGetMessage
01 USER32!GetMessageW
02 notepad!WinMain
03 notepad!WinMainCRTStartup
04 KERNEL32!BaseThreadInitThunk
05 ntdll!RtlUserThreadStart

1  Id: 984.eb8 Suspend: 1 Teb: 00007ff6`f411b000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2  Id: 984.1a8c Suspend: 1 Teb: 00007ff6`f4119000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

#  3  Id: 984.11b0 Suspend: 1 Teb: 00007ff6`f4117000 Unfrozen
# Call Site
00 ntdll!DbgBreakPoint
01 ntdll!DbgUiRemoteBreakin
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

The collection can be represented in a more compact form with multiplicities:

USER32!NtUserGetMessage
USER32!GetMessageW
notepad!WinMain           | 2* ntdll!NtWaitForWorkViaWorkerFactory | ntdll!DbgBreakPoint
notepad!WinMainCRTStartup | 2* ntdll!TppWorkerThread               | ntdll!DbgUiRemoteBreakin
4* KERNEL32!BaseThreadInitThunk
4* ntdll!RtlUserThreadStart

It can also be illustrated in the following diagram:

Such diagrams may help to spot Ubiquitous Components quickly.

Unified Stack Trace is also a generalization of Stack Trace Set where the latter only excludes fully duplicated stack traces but the former takes into account Constant Subtraces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -