Software Diagnostics Library

Metaphorical bijection from literary narratology to software narratology provides a pattern of Background and Foreground Components. This can be easily illustrated on pseudo-trace color diagrams. Suppose we troubleshoot a graphical issue using an ETW trace containing the output from all components of the problem system. Graphic components and their messages are foreground for a trace viewer (a person) against numerous background components (for example, database, file and registry access, shown in shades of green):

Trace viewers (for example, CDFAnalyzer) can filter out background component messages and present only foreground components (that I propose to call component foregrounding):

Of course, this process is iterative and parts of what once was foreground become background and candidates for further filtering:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a new open source tool similar to PDBFinder in functionality but with much simpler interface and internal implementation (based on the file name structure of the certain classes of TMF files for ETW). To be released this month on TraceAnalysis.org.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Incorrect Stack Trace pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Previously introduced Coupled Processes pattern involves an active request (or an action) and an active wait for a response (or the action status):

It is illustrated on this simple UML sequence diagram (process timeline represent collective request-response threads):

However, there is so called weak coupling when a process subscribes for notifications. Such threads most of the time are passive and processes are not blocked:

The coupling manifests itself when notifier threads start spiking CPU and bring their share of CPU consumption to the notified threads:

Here is an example of such threads from the forthcoming pattern interaction case study:

   5  Id: 61018.dbec Suspend: 1 Teb: 7ffae000 Unfrozen
ChildEBP RetAddr
01e3fa68 7c82787b ntdll!KiFastSystemCallRet
01e3fa6c 77c80a6e ntdll!NtRequestWaitReplyPort+0xc
01e3fab8 77c7fcf0 rpcrt4!LRPC_CCALL::SendReceive+0x230
01e3fac4 77c80673 rpcrt4!I_RpcSendReceive+0x24
01e3fad8 77ce315a rpcrt4!NdrSendReceive+0x2b
01e3fec0 771f4fbd rpcrt4!NdrClientCall2+0x22e
01e3fed8 771f4f60 winsta!RpcWinStationWaitSystemEvent+0x1c
01e3ff20 6582116c winsta!WinStationWaitSystemEvent+0x51
[...]
01e3ffec 00000000 kernel32!BaseThreadStart+0x34

In cases of synchronous notifications if a notified thread is blocked we have an instance of a reversed strong coupling.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Changed Environment pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Typical software narrative history consists of requests and responses, for example, function or object method calls and returns:

#     Module PID  TID  Time         File    Function Message
[...]
26060 dllA   1604 7108 10:06:21.746 fileA.c foo      Calling bar
[...]
26232 dllA   1604 7108 10:06:22.262 fileA.c foo      bar returns 0x5
[...]

The code that generates execution history is response-complete if it traces both requests and responses. For such code (except in cases where tracing is stopped before a response) the absence of expected responses could be a sign of blocked threads or quiet exception processing. The code that generates execution history is exception-complete if it also traces exception processing. Response-complete and exception-complete code is called call-complete. If we don’t see response messages for call-complete code we have Incomplete History.

In general, we can talk about the absence of certain messages in a trace as a deviation from the standard trace sequence template corresponding to a use case. The difference there is in a missing request too. This is a topic for next patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

per-PRCB queued, system-wide dispatcher, system-wide context swap and per-thread spinlocks (pp. 434-435)

physical over logical processor preference for scheduling (p. 435)

!smt command (p. 436) - here is the putput from x64 machine (from the output we can infer the following relationship logical processor -> core -> physical processor):

1: kd> !smt
SMT Summary:
------------
KeActiveProcessors: **-------------------------------------------------------------- (0000000000000003)
KiIdleSummary: -*-------------------------------------------------------------- (0000000000000002)
No PRCB SMT Set APIC Id
0 fffff80001991680 **-------------------------------------------------------------- (0000000000000003) 0x00000000
1 fffffa60005ec180 **-------------------------------------------------------------- (0000000000000003) 0x00000001

Maximum cores per physical processor: 2
Maximum logical processors per core: 1

NUMA (pp. 436 - 438) - I can see NUMA even on my small desktop system

1: kd> !numa
NUMA Summary:
------------
Number of NUMA nodes : 1
Number of Processors : 2
MmAvailablePages : 0x000C7CB9
KeActiveProcessors : (3)

NODE 0 (FFFFF80001995640):
ProcessorMask : (3)
Color : 0x00000000
MmShiftedColor : 0x00000000
Seed : 0x00000001
Right : 0x00000000
Left : 0x00000001
Zeroed Page Count: 0x0000000000000000
Free Page Count : 0x0000000000000000

Thread affinity (pp. 438 - 440) - see also Affine Thread crash dump analysis pattern: http://www.dumpanalysis.org/blog/index.php/2008/06/27/crash-dump-analysis-patterns-part-68/

uniprocessor flag as a workaround for multithreading defects (p. 439)

Set(Query)ProcessAffinityUpdateMode and dynamic processor changes (p. 442)

choosing a processor (idle ideal -> idle current -> idle previous -> current -> ideal running less priority thread) (pp. 433 - 444)

no guarantee to run all highest priority threads vs. always runs the highest priority thread (p. 444)

Advanced .NET Debugging by M. Hewardt:

value vs. reference types (p. 42)

sosex!bpsc (p. 46)

per frame managed stack trace: !ClrStack -a (p. 46)

d* for simple local value types, !dumpobj for references, !dumpvc for value type fields (pp. 46 - 47)

sync blocks (pp. 49 - 52) - here is the output from my x64 test program:

0:000> !ClrStack -a
OS Thread Id: 0x6e8 (0)

000000000013ed10 000007ff001ac709 System.IO.TextReader+SyncTextReader.ReadLine()
PARAMETERS:
this = 0x0000000002a2b568

0:000> !dumpobj 0x0000000002a2b568
Name: System.IO.TextReader+SyncTextReader
MethodTable: 000007feee67bea8
EEClass: 000007feedb851e0
Size: 32(0x20) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007feede86048 400018a 8 System.Object 0 instance 0000000000000000 __identity
000007feedecd198 4001c87 b18 System.IO.TextReader 0 shared static Null
>> Domain:Value 0000000000220840:0000000002a2b060 <<
000007feedecd198 4001c88 10 System.IO.TextReader 0 instance 0000000002a2af28 _in
ThinLock owner 1 (0000000000000000), Recursive 0

0:000> dq 0x0000000002a2b568-8
00000000`02a2b560 00000001`00000000 000007fe`ee67bea8
00000000`02a2b570 00000000`00000000 00000000`02a2af28
00000000`02a2b580 00000000`00000000 00000000`00000000
00000000`02a2b590 00000000`00000000 00000000`00000000
00000000`02a2b5a0 00000000`00000000 00000000`00000000
00000000`02a2b5b0 00000000`00000000 00000000`00000000
00000000`02a2b5c0 00000000`00000000 00000000`00000000
00000000`02a2b5d0 00000000`00000000 00000000`00000000

0:000> !syncblk 1
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
1 0000000000259bf8 0 0 0000000000000000 none 0000000002a28030 System.EventHandler
-----------------------------
Total 1
CCW 0
RCW 0
ComClassFactory 0
Free 0

thin sync blocks (p. 52)

Just got today a notification from one of corporate content providers that the following book was released a few months ago (reading it now and post a review later):

Technical Support Essentials: Advice to Succeed in Technical Support

It would be interesting to compare with the Art of Software Support book (I still hasn’t finished it yet).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After this 1st of April evening spent in circus with my family I approached a PC to check my blog (seen from the window) and found a number format exception on a side bar currency conversion widget:

At first I panicked but then recalled that I already encountered some currency conversion problems with that widget. At the time of this writing, it is no long a local April Fool’s day but the problem persists.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Deadlock (mixed objects, kernel space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A silicon based company announced a new product called GoToMyBug for remote viewing of bugs ……fff.fff.f.H.A…t….H..H%….u…A..H.EA.u.H……….ff.L..A…I..A.B..t-I..I.R.L..I..t.I….H..f…I…u.I..[.A…I..L..I……t.I…I..H…..f..H……H…I…u………fff.fff.ff.S…L..H..L..I..I.R…t…L..A…M..H…….I…u.I..[…%…..u.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There is similarity between the tandem of Memoidealism / Memorianity and Mimāṃsā school of philosophy. The former is based on hermeneutics of memory snapshots and the latter is based on hermeneutics of sacred texts. People often ask is there any God in Memory Religion. The answer is both yes and no. Like in Mimāṃsā system the gods are names found in memory snapshots. In that sense Memorianity has some features of sacramental atheism. At the same time Memory is the creator of memories.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Deadlock (LPC) pattern:

B/W

Color

 - Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Deadlock (mixed objects, user space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The philosophy of Melissus of Samos has the notion of an infinite number of moments in the past.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Deadlock (executive resources) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Impossibility to disable foreground after-wait priority boosts (p. 423)

CPU Stress tool (pp. 423 - 425, 428 - 430) - Good tool to model CPU spikes. See also Modeling CPU Spikes article I co-authored for Debugging Expert magazine

CPU starvation prevention via balance set manager thread (p. 427)

MMCSS priority boosts (p. 432)

Network throttling to prevent DPC activity interrupting MMCSS boosting (p. 433)

Advanced .NET Debugging by M. Hewardt:

System | shared | def app := bookkeeping, precreation | mscorlib | app code (pp. 37 - 38) - here we check that mscorlib assembly belongs to the shared domain:

0:003> !dumpdomain--------------------------------------System Domain: 000007fef00f8ef0LowFrequencyHeap: 000007fef00f8f38HighFrequencyHeap: 000007fef00f8fc8StubHeap: 000007fef00f9058Stage: OPENName: None--------------------------------------Shared Domain: 000007fef00f9860LowFrequencyHeap: 000007fef00f98a8HighFrequencyHeap: 000007fef00f9938StubHeap: 000007fef00f99c8Stage: OPENName: NoneAssembly: 00000000003a2d10————————————–Domain 1: 0000000000390840LowFrequencyHeap: 0000000000390888HighFrequencyHeap: 0000000000390918StubHeap: 00000000003909a8Stage: OPENSecurityDescriptor: 00000000003930e0Name: TestCLR.exe

[...]

Assembly: 00000000003a2d10[C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]ClassLoader: 00000000003a2dd0SecurityDescriptor: 00000000003a2110Module Name000007feeda51000 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

0:003> !dumpassembly 00000000003a2d10Parent Domain: 000007fef00f9860Name: C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dllClassLoader: 00000000003a2dd0SecurityDescriptor: 000000000335db78Module Name000007feeda51000 C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

Multimodule assemblies with separate PE file for a manifest (p. 40)

Today we introduce an icon for Deadlock (critical sections) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Dr. DebugLove is working on a new album release scheduled for April. Its title is simply “Computation”. MP3 download will be available too. Stay tuned to the waves of computation and don’t forget to join Facebook fan group: Music of Computation

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -