Software Diagnostics Library

Indexical Trace pattern describes an Inter-Correlation pattern variant when we have a trace that has messages of interest pointing to specific activity regions in another trace. The latter trace can be very huge, from another computer and split into many parts (Split Trace). This pattern is very helpful when the problem needs to be diagnosed in the large split trace but we don’t know when it happened. Then an index trace that may have recorded software execution account (for example, in the case of a broker-like architecture) and can point to the right trace fragment from the split trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Static program analysis is used to eliminate certain coding errors that may lead to abnormal software behaviour. So it is naturally a part of software diagnostics but at source code level. Our goal here is to identify certain patterns directly linkable to patterns we see in memory dumps and software logs and collect them into a catalog. One such pattern candidate is called Loop Construct. It covers conditional and unconditional loops, for example, in one of modern languages:

extern bool soonToBeTrue; 

int mediumValue = ...;

while (true)

{

  TRACE("Waiting");

  sleep(mediumValue);

  if (soonToBeTrue)

  {

    break;

  }

  doHeavyWork();

}

while (--pControl->aFewPasses)

{

  TRACE("Waiting");

  sleep(mediumValue);

  doHeavyWork();

}

Such loops may potentially lead to Spiking Thread memory dump analysis and High Message Current and Density trace analysis patterns. Of course, we shouldn’t suspect every loop but only some that have potential to be altered by Local Buffer Overflow (for mediumValue) or Shared Buffer Overwrite (for Control.aFewPasses) or by a race condition (soonToBeTrue).

We expect things to get more interesting when we start associating source code that uses certain API with patterns of abnormal behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If you need to get various real life software traces with millions of lines from complex software environments to learn pattern-driven software log analysis you can us Process Monitor as a modeling tool. Here you can abstract from their “monitoring” and API interception context and consider trace messages as emitted from various processes and threads (like Citrix CDF traces). This approach was used in Accelerated Windows Software Trace Analysis training and Debugging TV Frames episode 0×19.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Before deciding on whether to retrospect on 2012 we found that since March 14, 2008 this site has had more than 1 million visitors with more than 33% returning. So instead of just 2012 we decided to retrospect on that interval up to December 31, 2012. Google Analytics has improved since last January, 2012 and now made our task easier. So we start with the first one hundred sites referring to us:

Next table is distribution of visits among countries:

Then the first 100 network locations:

Then the first 100 search keywords and phrases that led to us:

Then browser stats (we have never thought that there are so many of them):

Browser	Visits
Internet Explorer	446051
Firefox	356686
Chrome	184535
Opera	45787
Safari	24123
Mozilla	3780
Mozilla Compatible Agent	2401
Android Browser	1337
Konqueror	1057
IE with Chrome Frame	982
Opera Mini	705
SeaMonkey	503
Safari (in-app)	197
Lunascape	144
BlackBerry8900	128
Camino	126
RockMelt	124
(not set)	96
Netscape	72
Playstation 3	36
IUC	34
Googlebot	29
Lynx	24
Unsupported Browser Version	22
BlackBerry9630	21
NetFront	17
BlackBerry9700	15
Microsoft-Symbol-Server	14
BlackBerry9000	12
Galeon	11
Midori	9
NokiaE63	9
Yahoo! Slurp	9
BlackBerry8530	8
BlackBerry8520	7
PagePeeker.com	7
SAMSUNG-SGH-I617	7
BlackBerry9530	6
JUC	6
MSR-ISRCCrawler	6
OpenWave	6
anonimo	5
BlackBerry9300	5
HTC_HD2_T8585 Opera	5
Nokia5233	5
Space Bison	5
-Vasya	4
Blazer	4
Uzbl	4
-^_^- Hello	3
<?echo ‘<pre>’; system	3
12345	3
BlackBerry9330	3
BlackBerry9650	3
HTC_P3700 Opera	3
HTC_TyTN_II Mozilla	3
NOKIAN78	3
Playstation Portable	3
PPC; 240×320; HTC_P3450	3
undefined GoogleToolbarBB	3
anonymous	2
Empty	2
GreatBrowse	2
Helyi user agent	2
HTC_Touch_Pro2_T7373 Opera	2
HTC_Touch2_T3333 Opera	2
J2ME	2
Mozilla 5.0	2
NokiaC1-01	2
NokiaC3-00	2
NokiaC7-00	2
NokiaX2-01	2
nwzfq	2
test	2
—	1
?M5	1
“PagePeeker.com”	1
<?include	1
<script>alert	1
<SCRIPT>window.location=’http:	1
2.0.0.11	1
31337′	1
8900b	1
AltaVista Intranet V2.0 evreka.com crawler@evreka.com	1
annoying	1
AppEngine-Google;	1
BlackBerry9500	1
BlackBerry9550	1
bwh3_user_agent	1
Citrix	1
EBABrowser	1
EY	1
fake_user_agent Mozilla	1
FAST Enterprise Crawler 6 used by Reed Exhibitions	1
foo	1
General Browser	1
GOOGLEBOT	1
HD_mini_T5555 Opera	1
Hellbrowser 6.66	1
holy_teacher FirePHP	1
HTC_P3490 Opera	1
HTC_P4550 Mozilla	1
HTC_Polaris Mozilla	1
HTC_Touch_3G_T3232 Opera	1
HTC_Touch_HD_T8282 Opera	1
HTC_Touch_Pro_T7272 Opera	1
HTC_Touch2_T3320 Opera	1
HTC-8900	1
IE 8	1
IE6	1
iTunes	1
Keep Out	1
KraftwayBrowser2	1
Links	1
Maemo Browser	1
Medusa	1
MERONG	1
Motorola_ES405B	1
mozilla	1
Mozilla Firefox	1
MS-OC 4.0	1
msie	1
NCSA Mosaic	1
NightDynamo AdminPanel v0.2.1	1
Nokia2700c-2	1
Nokia2730c-1	1
Nokia305	1
Nokia5230	1
Nokia5310XpressMusic	1
Nokia5800 XpressMusic	1
Nokia6300	1
Nokia6700c-1	1
NokiaC2-01	1
NokiaC2-02	1
NokiaC2-03	1
NokiaC5-03	1
nokiac6-00	1
NokiaC6-00	1
NOKIAE65	1
NokiaE66	1
NokiaE71	1
NokiaE71-2;Mozilla	1
NokiaE72-1	1
NokiaN-GageQD	1
NokiaN70-1	1
NokiaNokia 6210s	1
NoneOfYourBusiness	1
nothisname_wangxiaoyang3	1
OmniWeb	1
Palm750	1
Peeplo Screenshot Bot	1
PerTrUsTsQuiD	1
pippos.7	1
PPC; 480×800; HTC_Touch_HD_T8282; OpVer 34.159.1.612	1
PriceGoblin User Agent	1
Private	1
Privoxy	1
Read Later	1
SAMSUNG-GT-E2222	1
samsung-gt-s3653	1
samsung-gt-s3653 UNTRUSTED	1
SAMSUNG-S8000	1
SAMSUNG-SGH-I637	1
Samsung-SPHM540 Polaris	1
SmallProxy 3.5.4	1
SonyEricssonK750	1
Surf	1
tdhbrowser	1
TiFiC Client Z	1
union update table sd_users set userid=9 where username=’coco	1
unknown	1
Unknown	1
UNTRUSTED	1
Updownerbot	1
WIN	1
WinXP SP2	1
Wlwap	1
WM5 PIE	1
Xda_orbit_2; 240×320	1
Xyi znat kakoi browser MRA 5.7	1
ZooShot 0.1a	1
ZooShot 0.42	1

and finally mobile devices stats (you may find your own device there):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This pattern is called Opposition Messages (borrowed from binary opposition originated in Saussure’s structuralism). It covers the following pairs of messages usually found in software traces and logs such as:

• open / close
• create / destroy
• allocate / free (deallocate)
• call / return
• enter / exit (leave)
• load / unload
• save / load
• lock / unlock
• map / unmap

The absence of an opposite may point to some problems such as synchronization and leaks or Incomplete History (wait chains). There can always be a possibility that a second term is missing due to Sparse Trace but this is a poor implementation choice that leads to confusion during troubleshooting and debugging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Some tracing tools such as Citrix CDFControl have an option to split software traces and logs into several files during long recording. Although this should be done judiciously it is really necessary sometimes. We call this pattern Split Trace. What to do if we get several trace files and we want to use some other analysis tool such as Citrix CDFAnalyzer? If we know that the problem happened just before the tracing was stopped we can look at the last few such files from the file sequence (although we recommend a circular trace here). Otherwise we can convert them into CVS files and import into Excel which also supports adjoint threading.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

To celebrate the last unique occurrence of 12.12.12 in this Century I created a Computical Art (Computicart) picture that uses minimalist software trace analysis diagrams I use for trace and log analysis patterns made even better for Accelerated Windows Software Trace Analysis course. The following diagram symbolizes several patterns. Suppose we trace personal or world events each year and form an Adjoint Thread with messages having DD=MM=YY timestamp invariant (or filter them). Then we clearly see a Discontinuity before the next Century with much bigger Time Delta than between such messages. Also these messages form a Periodic Message Block in relation to the full trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We have decided to factor out Anchor Messages example of message interleave into another pattern called Message Interleave. It covers superposition of different anchor messages, for example, process launch and exit, or DLL load and unload:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we don’t see anything in the trace or see very little because particular source code fragment was not covered by trace statements (see also PLOTs):

We call this pattern Sparse Trace and this is different from Missing Component pattern where some modules were not included for tracing explicitly although there is tracing code there or Visibility Limit pattern where tracing is intrinsically impossible. Often technical support and escalation engineers request to add more trace statements and software engineers extend tracing coverage iteratively as needed.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I get many questions on whether software log analysis patterns from Software Diagnostics Institute are OS or platform or product specific. My answer is that they are independent from all of them because they are based on viewing software logs as stories of computation and were discovered by application of narratological analysis (software narratology). In addition to these patterns there exist domain specific problem patterns such as wrong hotfix level or specific product error code during software installation or execution. Typical examples of support for such platform and product specific type of patterns include Microsoft Windows Problem Reporting and Citrix Auto Support.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Most of the time it is not possible to trace from the beginning. Obviously, internal application tracing cannot trace anything before that application start and its early initialization. The same is for system wide tracing which cannot trace before the tracing subsystem or service starts. Therefore, each log has its Visibility Limit in addition to possible truncation or missing components:

One of solutions would be to use different tracing tools and inter-correlation to glue activities, for example, Process Monitor and CDFControl.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Doing memory dump analysis training for more than 2 years I found that students are divided into 2 types: those who prefer to see source code first and those who want to see a memory dump first. We actually prefer to show a memory dump first and then explore it to find certain patterns of abnormal structure and behavior. Software Diagnostics Services used this approach to design its Accelerated Windows Memory Dump Analysis and Accelerated .NET Memory Dump Analysis courses. Students explore memory dumps and debugger logs to find memory dump analysis patterns which are introduced when necessary. After that they can check source code of modeling applications if they have development experience. Accelerated Windows Software Trace Analysis course uses a different approach. It introduces all software trace analysis patterns at once because they are patterns from software narratology independent from programming languages and software platforms. After that they explore and analyze software traces and logs. We can summarize these 2 approaches on this diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Relative Density pattern describes anomalies related to the semantically related pairs of trace messages, for example, “data arrival” and “data display”. Their statement densities can be put in a ratio (also called specific gravity) and compared between working and non-working scenarios. Because the total number of trace messages cancel each other we have just mutual ratio of two message types. In our hypothetical “data” example the increased ratio of “data arrival” to “data display” messages accounts for reported visual data loss and sluggish GUI.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If trace messages contain some character or formatted data that is passed from module to module or between threads and processes it is possible to trace that data and form a Data Flow thread similar to an adjoint thread we have when we filter by a specific message. However, in the former case we have different message types.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Empty Trace is another trivial missing pattern that we need to add to make software log analysis pattern system complete. It ranges from a totally empty trace message list where only a meta-trace header (if any) describing overall trace structure is present to a few messages where we expect thousands. This is also an extreme case of Truncated Trace, No Activity and Missing Component patterns for a trace taken as a whole. Note, that an empty trace file doesn’t necessarily have a zero file size because a tracing architecture may preallocate some file space for block data writing.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

While working on Accelerated Windows Software Trace Analysis training I discovered some missing patterns needed for completeness despite their triviality. One of them is called Error Message. Here an error is reported either explicitly (”operation failed”) or implicitly as an operation status result such as 0xC00000XX. Sometimes, a trace message designer specifies that the number value was supplied for information only and should be ignored. Some error messages may contain information that is not relevant for the current software incident, the so called false positive errors. Some tracing architectures and tools include message information category for errors, such as Citrix CDF (ETW-based) where you filter by error category to get an adjoint thread. Note, that the association of a trace statement with an error category is left at the discretion of an engineer writing code and information category messages may contain implicit errors such as last error and return status reports.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The case of an error reporting fault chain led me to First Fault Stack Trace memory dump analysis pattern that corresponds to First Fault software diagnostics pattern proper. Here the term first fault is used for an exception that was either ignored by surrounding code or led to other exceptions or error message boxes with stack traces that masked the first one. Typical examples where it is sometimes possible to get a first exception stack trace include but not limited to:

• Fault in error reporting started as a fault in in some other process
• Hidden exception in user or kernel space
• Double fault
• Nested exception in unmanaged and managed code
• Nested offender

It is also sometimes possible unless a stack region was paged out to get partial stack traces from execution residue when the sequence of return addresses was partially overwritten by subsequently executed code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Periodic Message Block is similar to Periodic Error but not limited to errors or failure reports. One such example we recently encountered is when some adjoint activity (such as messages from specific PID) stop to appear after the middle of the trace and after that there are repeated blocks of similar messages from different PIDs with their threads checking for some condition (waiting for event and reporting timeouts):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Accelerated Windows Software Trace Analysis Training.

These are important meta-patterns of monitoring and software problem analysis reports. It is often the case that we have software artifacts and some problem description and we need to provide recommendations on further troubleshooting. Most of the time such an analysis and associated response fit into abstract schemes where we can just substitute variables for concrete states, actions, artifacts and behavioral descriptions. Sometimes we also have difficulty to write such analysis reports so we hope report schemes is of help here to provide organizing templates for thought process. The first such scheme we call Missing Cause Trace:

1. If Action then Behavior
2. We have a trace of Behavior
3. We need a trace of Action and Behavior

The difference with Truncated Trace pattern here is that in a truncated trace it was intended to trace certain behaviour but the tracing session was stopped prematurely or started too late. In a missing cause trace only a part of necessary activity was traced and the missing part wasn’t thought of or considered for tracing.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The modern software trace recording, visualization and analysis tools such as Process Monitor, Xperf, WPR and WPA provide stack traces associated with trace messages. Consider stack traces as software traces we have, in a more general case, traces (fibers) bundled together on (attached to) a base software trace. For example, a trace message, that mentions an IRP can have its I/O stack attached together with thread stack trace with function calls leading to a function that emitted the trace message. Another example is association of different types of traces with trace messages such as managed and unmanaged ones. This general trace analysis pattern needs a name so we opted for Fiber Bundle as analogy with a fiber bundle from mathematics. Here’s a graphical representation of stack traces recorded for each trace message where one message also has an associated I/O stack trace:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -