Archive for June, 2014

Trace Analysis Patterns (Part 89)

Thursday, June 26th, 2014

Usually when we analyse traces and find an Anchor or Error Message we do backtracking using a combination of Data Flow and Message Sets and selecting appropriate log messages to form Back Trace leading to a possible root cause message:

This pattern is different from Error Thread pattern which just backtracks messages having the same TID (or in general ATID). It is also different from Exception Stack Trace pattern which is just a serialized stack trace from memory snapshot.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 88)

Tuesday, June 24th, 2014

The previous patterns such as Basic Facts and Vocabulary Index address the mapping of a problem description to software execution artefacts such traces and logs. Indirect Facts analysis pattern addresses the problem of an incomplete problem description. However, we need another pattern for completeness that addresses the mapping from a log to troubleshooting and debugging recommendations. We call it Hidden Facts which are uncovered by trace analysis. Of course, there can be many such hidden facts and usually they are uncovered after narrowing down analysis to particular Threads of Activity, Adjoint Threads, Message Context, Message Set, or Data Flow. The need for that pattern had arisen during the pattern-oriented analysis of the trace case study from Malcolm McCaffery and can be illustrated on this diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 208)

Monday, June 23rd, 2014

When we suspect a particular thread doing I/O but IRP is missing in the output of !thread WinDbg command the best way is to examine the list of IRPs and associated threads from the output of !irpfind command. Here is a synthesized example from a few Virtualized Young System crash dumps:

0: kd> !thread fffffa8004e2d280

THREAD fffffa8004e2d280 Cid 0004.0020 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
fffff880009ec440 NotificationEvent
Not impersonating
[…]

0: kd> !irpfind

Irp [ Thread ] irpStack: (Mj,Mn) DevObj [Driver] MDL Process
[...]
fffffa800424e4e0 [fffffa8004e2d280] irpStack: (3, 0) fffffa8004ed6d40 [ \Driver\DriverA]
[…]

Now we can inspect the found IRP (!irp command) and device object (for example, by using !devobj and !devstack commands). Sometimes we can see the same IRP address as Execution Residue among “Args to Child” values in the output of !thread command or kv (if the thread is current). We call such pattern Hidden IRP.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 207)

Saturday, June 21st, 2014

The pattern called Small Value deals with easily recognizable values such as handles, timeouts, mouse pointer coordinates, enumeration values, window messages, etc. There is another kind of values we call Design Values, for example, 256 (+/- 1) or some other round value. Here we can also add some regular patterns in hex representation such as window handles or flags, for example, such as 0×10008000. Such designed values may fall into some module range too, the so called Coincidental Symbolic Information pattern. If we see a design value in the output of WinDbg commands especially related to abnormal behaviour patterns, not necessarily as a stack trace parameter, which can be False, then it might point to some design limitations that were reached. For example, Blocked ALPC Queue may have a limitation on I/O Completion Port when we have ALPC Wait Chains in an unresponsive system:

0: kd> !alpc /p <port_address>
[...]
512 thread(s) are registered with port IO completion object:
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -