Software Diagnostics Library

This pattern is similar to No Component Symbols memory analysis pattern and is called No Trace Metafile:

#     Module    PID  TID  Time        Message
21372 \src\dllA 2968 5476 3:55:10.004 Calling foo()
21373 Unknown   2968 5476 3:55:10.004 ????? Unknown( 27): GUID=1EF56EBD-A7FC-4892-8DBA-00AD813F8A24 (No Format Information found).
21374 Unknown   2968 5476 3:55:10.004 ????? Unknown( 27): GUID=1EF56EBD-A7FC-4892-8DBA-00AD813F8A24 (No Format Information found). 
21375 Unknown   2968 5476 3:55:10.004 ????? Unknown( 27): GUID=1EF56EBD-A7FC-4892-8DBA-00AD813F8A24 (No Format Information found).
21376 Unknown   2968 5476 3:55:10.004 ????? Unknown( 28): GUID=1EF56EBD-A7FC-4892-8DBA-00AD813F8A24 (No Format Information found).
21377 Unknown   2968 5476 3:55:10.004 ????? Unknown( 23): GUID=1EF56EBD-A7FC-4892-8DBA-00AD813F8A24 (No Format Information found).
21378 \src\dllA 2968 5476 3:55:10.004 Calling bar()

In some case when we don’t have TMF files it is possible to detect broad behavioral patterns such as:

• Circular Trace
• Statement Density and Current
• Discontinuity
• Time Delta
• Trace Acceleration

By looking at Thread of Activity we can also sometimes infer the possible component name based on surrounding trace messages with present TMF files, escpecially when we have source code access. For example, in the trace above it can be dllA or any other module that foo function calls.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Waiting Thread Time (kernel dumps) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

My drive to generalization led me to place an adornment on the portal to highlight the fact that memory and software trace analysis patterns are under an umbrella of general software behaviour patterns:

http://www.dumpanalysis.org/Software-Behavior-Patterns-Headline

In the forthcoming post series I plan to write about similarities between these two branches and also provide pattern examples from non-Windows platforms. All this material will provide the foundation for the forthcoming book Software Behavior: A Guide to Systematic Analysis (ISBN: 978-1906717162).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Previously announced book is now available on Amazon:

Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoidealism explains God as Inaccessible Memory (Memory Region A on the picture below). It means that Memory Region B doesn’t have pointers that point outside of it. In other words, all operating fields of Region B pointers are in Region B. However, Region A can have pointers pointing to Region B and modify it, effectively producing a miracle as perceived by Region B. In other words, some perception field links of Region B pointers may come from outside. You can object that Region B can have deeper pointers by making their memory locations bigger (from N-bit to 2*N-bit for example) but Memory may not be flat, discrete and bit-like. For one possible Christian interpretation we can replace Region B with the heavens and the earth (Genesis 1.1) and replace Region A with God-Creator (note also that in Memorianity Memory creates memories [notes on memoidealism 1.10]).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes when a high number of interrupts is reported but there are no signs of an interrupt storm or pending DPCs in a memory dump file it is useful to search for Hardware Activity in running and / or suspected threads. This can be done by examining execution residue left on a thread raw stack. Although found driver activity might not be related to reported problems it can be a useful start for driver elimination procedure for the general recommendation to check drivers for any updates. Here is an example of a thread raw stack with a network card doing “Scatter-Gather” DMA (more extensive example is coming in a separate pattern cooperation case study):

1: kd> !thread
THREAD f7732090 Cid 0000.0000 Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 1
Not impersonating
Owning Process 8089db40 Image: Idle
Attached Process N/A Image: N/A
Wait Start TickCount 0 Ticks: 24437545 (4:10:03:56.640)
Context Switch Count 75624870
UserTime 00:00:00.000
KernelTime 4 Days 08:56:05.125
Stack Init f78b3000 Current f78b2d4c Base f78b3000 Limit f78b0000 Call 0
Priority 0 BasePriority 0 PriorityDecrement 0
ChildEBP RetAddr Args to Child
f3b30c5c 00000000 00000000 00000000 00000000 LiveKdD+0x1c07

1: kd> dds f78b0000 f78b3000
f78b0000 00000000
f78b0004 00000000
f78b0008 00000000
f78b000c 00000000
f78b0010 00000000
[...]
f78b2870 8b3de0d0
f78b2874 80887b75 nt!KiFlushTargetSingleTb+0xd
f78b2878 8b49032c
f78b287c 00000000
f78b2880 2d003202
f78b2884 00000000
f78b2888 00000000
f78b288c 2d003202
f78b2890 8b490302
f78b2894 f78b28a4
f78b2898 80a61456 hal!KfLowerIrql+0x62
f78b289c 2d00320a
f78b28a0 00000000
f78b28a4 8b3de0d0
f78b28a8 8b3e3730
f78b28ac 00341eb0
f78b28b0 f78b2918
f78b28b4 f63fbf78 NetworkAdapterA!SendWithScatterGather+0×318
f78b28b8 8b3de0d0
f78b28bc 8b341eb0
f78b28c0 f78b28d4
f78b28c4 00000000
f78b28c8 80a5f3c0 hal!KfAcquireSpinLock
f78b28cc 00000000
f78b28d0 8b3de0d0
f78b28d4 00000000
f78b28d8 8b3de0d0
f78b28dc 8b3eb730
f78b28e0 005a7340
f78b28e4 f78b294c
f78b28e8 f63fbf78 NetworkAdapterA!SendWithScatterGather+0×318
f78b28ec 8b3de0d0
f78b28f0 8a5a7340
f78b28f4 f78b2908
f78b28f8 00000000
f78b28fc 8b3de0d0
f78b2900 8b0f5158
f78b2904 001e2340
f78b2908 f78b2970
f78b290c f63fbf78 NetworkAdapterA!SendWithScatterGather+0×318
f78b2910 8b3de0d0
f78b2914 8b1e2340
f78b2918 f78b292c
f78b291c 00000000
f78b2920 80a5f3c0 hal!KfAcquireSpinLock
f78b2924 00000000
f78b2928 8b3de0d0
f78b292c 00000000
f78b2930 8b3eb700
f78b2934 00000000
f78b2938 00000000
f78b293c 00000000
f78b2940 00000000
f78b2944 00000000
f78b2948 00000000
f78b294c 0a446aa2
f78b2950 f78b29b8
f78b2954 8b0f5158
f78b2958 8b01ce10
f78b295c 00000001
f78b2960 8b3de0d0
f78b2964 80a5f302 hal!HalpPerfInterrupt+0×32
f78b2968 00000001
f78b296c 8b3de0d0
f78b2970 80a5f302 hal!HalpPerfInterrupt+0×32
f78b2974 8b3de302
f78b2978 f78b2988
f78b297c 80a61456 hal!KfLowerIrql+0×62
f78b2980 80a5f3c0 hal!KfAcquireSpinLock
f78b2984 8b3de302
f78b2988 f78b29a4
f78b298c 80a5f44b hal!KfReleaseSpinLock+0xb
f78b2990 f63fbbbf NetworkAdapterA!SendPackets+0×1b3
f78b2994 8a446a90
f78b2998 8b0e8ab0
f78b299c 00000000
f78b29a0 008b29d0
f78b29a4 f78b29bc
f78b29a8 f7163790 NDIS!ndisMProcessSGList+0×90
f78b29ac 8b3de388
f78b29b0 f78b29d0
f78b29b4 00000001
f78b29b8 00000000
f78b29bc f78b29e8
f78b29c0 80a60147 hal!HalBuildScatterGatherList+0×1c7
f78b29c4 8b0e89b0
f78b29c8 00000000
f78b29cc 8a44cde8
f78b29d0 8b1e2340
f78b29d4 8a446aa2
f78b29d8 8b026ca0
f78b29dc 8b1e2340
f78b29e0 8b0e8ab0
f78b29e4 8b0e8ab0
f78b29e8 f78b2a44
f78b29ec f716369f NDIS!ndisMAllocSGList+0xda
f78b29f0 8a44cde8
f78b29f4 8b0e89b0
f78b29f8 8a446a70
f78b29fc 00000000
f78b2a00 00000036
f78b2a04 f7163730 NDIS!ndisMProcessSGList
f78b2a08 8b1e2340
f78b2a0c 00000000
f78b2a10 8a44cde8
f78b2a14 00000218
f78b2a18 8b1e2308
f78b2a1c 00000103
f78b2a20 8b0e8ab0
f78b2a24 8a446a70
f78b2a28 8a44cde8
f78b2a2c 00000036
f78b2a30 8b0e8ab0
f78b2a34 00000036
f78b2a38 00000000
f78b2a3c 00000000
f78b2a40 029a9e02
f78b2a44 f78b2a60
f78b2a48 f71402ff NDIS!ndisMSendX+0×1dd
f78b2a4c 8b490310
f78b2a50 8b1e2340
f78b2a54 8a446a70
f78b2a58 8a9a9e02
f78b2a5c 8a9a9e02
f78b2a60 f78b2a88
f78b2a64 f546c923 tcpip!ARPSendData+0×1a9
f78b2a68 8b3e76c8
f78b2a6c 8b1e2340
f78b2a70 8a9a9ea8
f78b2a74 8b490310
f78b2a78 80888b00 nt!RtlBackoff+0×68
f78b2a7c 8a446a70
f78b2a80 8a446aa2
f78b2a84 8a446a70
f78b2a88 f78b2ab4
f78b2a8c f546ba5d tcpip!ARPTransmit+0×112
f78b2a90 8b490310
f78b2a94 8b1e2340
f78b2a98 8a9a9ea8
f78b2a9c 00000103
f78b2aa0 8a446a70
f78b2aa4 00000000
f78b2aa8 8b342398
f78b2aac 8a47e1f8
f78b2ab0 8b1e2340
f78b2ab4 f78b2bf0
f78b2ab8 f546c4fc tcpip!_IPTransmit+0×866
f78b2abc 8a9a9ebc
f78b2ac0 f78b2b02
f78b2ac4 00000001
[…]

We also do a sanity check for coincidental symbols:

1: kd> ub f63fbf78
NetworkAdapterA!SendWithScatterGather+0x304:
f63fbf64 push    eax
f63fbf65 push    edi
f63fbf66 push    esi
f63fbf67 mov     dword ptr [ebp-44h],ecx
f63fbf6a mov     dword ptr [ebp-3Ch],ecx
f63fbf6d mov     dword ptr [ebp-34h],ecx
f63fbf70 mov     dword ptr [ebp-2Ch],ecx
f63fbf73 call    NetworkAdapterA!PacketRetrieveNicActions (f63facd2)

1: kd> ub f63fbbbf
NetworkAdapterA!SendPackets+0x190:
f63fbb9c cmp     dword ptr [esi+0Ch],2
f63fbba0 jl      NetworkAdapterA!SendPackets+0x19e (f63fbbaa)
f63fbba2 mov     dword ptr [ecx+3818h],eax
f63fbba8 jmp     NetworkAdapterA!SendPackets+0x1a4 (f63fbbb0)
f63fbbaa mov     dword ptr [ecx+438h],eax
f63fbbb0 mov     dl,byte ptr [esi+2BCh]
f63fbbb6 mov     ecx,dword ptr [ebp+8]
f63fbbb9 call    dword ptr [NetworkAdapterA!_imp_KfReleaseSpinLock (f640ca18)]

1: kd> ub 80a60147
hal!HalBuildScatterGatherList+0x1b0:
80a60130 je      hal!HalBuildScatterGatherList+0x1b9 (80a60139)
80a60132 mov     dword ptr [eax+4],1
80a60139 push    dword ptr [ebp+20h]
80a6013c push    eax
80a6013d mov     eax,dword ptr [ebp+0Ch]
80a60140 push    dword ptr [eax+14h]
80a60143 push    eax
80a60144 call    dword ptr [ebp+1Ch]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a variation of Hooked Functions pattern for kernel space. In addition to trampoline patching we also see a modified service table:

0: kd> !chkimg -lo 50 -d !nt -v
Searching for module with expression: !nt
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\ntkrnlmp.exe\4B7A8E62280000\ntkrnlmp.exe
No range specified

Scanning section:    .text
Size: 625257
Range to scan: 80801000-80899a69
    808373e3-808373e9  7 bytes - nt!KeAcquireQueuedSpinLockAtDpcLevel+1b
 [ f7 41 04 01 00 00 00:e9 00 0d b2 76 cc cc ]
    8083e6c8-8083e6cb  4 bytes - nt!KiServiceTable+440 (+0×72e5)
 [ 98 4e 98 80:d0 66 e9 f4 ]
    80840605-8084060a  6 bytes - nt!KxFlushEntireTb+9 (+0×1f3d)
 [ ff 15 1c 10 80 80:e9 a5 7a b1 76 cc ]
Total bytes compared: 625257(100%)
Number of errors: 17

Scanning section: MISYSPTE
Size: 1906
Range to scan: 8089a000-8089a772
Total bytes compared: 1906(100%)
Number of errors: 0

Scanning section:   POOLMI
Size: 7868
Range to scan: 8089b000-8089cebc
Total bytes compared: 7868(100%)
Number of errors: 0

Scanning section: POOLCODE
Size: 7754
Range to scan: 8089d000-8089ee4a
Total bytes compared: 7754(100%)
Number of errors: 0

Scanning section:     PAGE
Size: 1097281
Range to scan: 808bc000-809c7e41
Total bytes compared: 1097281(100%)
Number of errors: 0

Scanning section:   PAGELK
Size: 63633
Range to scan: 809c8000-809d7891
Total bytes compared: 63633(100%)
Number of errors: 0

Scanning section:  PAGEWMI
Size: 7095
Range to scan: 809ef000-809f0bb7
Total bytes compared: 7095(100%)
Number of errors: 0

Scanning section:   PAGEKD
Size: 16760
Range to scan: 809f1000-809f5178
Total bytes compared: 16760(100%)
Number of errors: 0

Scanning section: PAGEHDLS
Size: 7508
Range to scan: 809f7000-809f8d54
Total bytes compared: 7508(100%)
Number of errors: 0
17 errors : !nt (808373e3-8084060a)

0: kd> dds 8083e6c8
8083e6c8  f4e966d0 DriverA+0×20d8
8083e6cc  80983436 nt!NtUnloadKey2
8083e6d0  809837b5 nt!NtUnloadKeyEx
8083e6d4  8091cec8 nt!NtUnlockFile
8083e6d8  80805d80 nt!NtUnlockVirtualMemory
8083e6dc  80937630 nt!NtUnmapViewOfSection
8083e6e0  808e7154 nt!NtVdmControl
8083e6e4  809c6ba3 nt!NtWaitForDebugEvent
8083e6e8  8092dc24 nt!NtWaitForMultipleObjects
8083e6ec  8092ccf4 nt!NtWaitForSingleObject
8083e6f0  809c132f nt!NtWaitHighEventPair
8083e6f4  809c12c3 nt!NtWaitLowEventPair
8083e6f8  80925c8d nt!NtWriteFile
8083e6fc  80901790 nt!NtWriteFileGather
8083e700  8091214c nt!NtWriteRequestData
8083e704  8093e63b nt!NtWriteVirtualMemory
8083e708  80822751 nt!NtYieldExecution
8083e70c  808c7c46 nt!NtCreateKeyedEvent
8083e710  8093eee3 nt!NtOpenKeyedEvent
8083e714  809c1ee8 nt!NtReleaseKeyedEvent
8083e718  809c2183 nt!NtWaitForKeyedEvent
8083e71c  809a610b nt!NtQueryPortInformationProcess
8083e720  809a6123 nt!NtGetCurrentProcessorNumber
8083e724  809a1849 nt!NtWaitForMultipleObjects32
8083e728  90909090
8083e72c  1c0d3b90
8083e730  0f8089f1
8083e734  037aaa85
8083e738  00c1f700
8083e73c  0fffff00
8083e740  037a9e85
8083e744  9090c300

0: kd> u 808373e3
nt!KeAcquireQueuedSpinLockAtDpcLevel+0×1b:
808373e3 jmp     DriverB+0×10e8 (f73580e8)
808373e8 int     3
808373e9 int     3
808373ea je      nt!KeAcquireQueuedSpinLockAtDpcLevel+0×12 (808373da)
808373ec pause
808373ee jmp     nt!KeAcquireQueuedSpinLockAtDpcLevel+0×1b (808373e3)
nt!KeReleaseInStackQueuedSpinLockFromDpcLevel:
808373f0 lea     ecx,[ecx]
nt!KeReleaseQueuedSpinLockFromDpcLevel:
808373f2 mov     eax,ecx

0: kd> u 80840605
nt!KxFlushEntireTb+0×9:
80840605 jmp     DriverB+0×10af (f73580af)
8084060a int     3
8084060b mov     byte ptr [ebp-1],al
8084060e mov     ebx,offset nt!KiTbFlushTimeStamp (808a7100)
80840613 mov     ecx,dword ptr [nt!KiTbFlushTimeStamp (808a7100)]
80840619 test    cl,1
8084061c jne     nt!KxFlushEntireTb+0×19 (8082cd8d)
80840622 mov     eax,ecx

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Truncated Dump pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Some recent news about StressPrinters tool designed according to Tool Façade DebugWare pattern:

“HP tests its print drivers with the StressPrinters tool provided by Citrix to simulate a user logon where multiple printers are autocreated concurrently.”

Source

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Managed Code Exception pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Befind every trace and its messages is source code:

Borrowing the acronym PLOT (Program Lines of Trace) we now try to discern basic source code patterns that give rise to simple message patterns in software traces. There are only a few distinct PLOTs and the ability to mentally map trace statements to source code is crucial to software trace reading and comprehension. More about that in subsequent parts. More complex message patterns (for example, specific message blocks or correlated messages) arise from supportable and maintainable realizations of architectural, design and implementation patterns and will be covered in another post series.

I was thinking about acronym SLOT (Source Lines of Trace) but decided to use PLOT because it metaphorically bijects into literary theory and narrative plots.

Forthcoming CDF and ETW Software Trace Analysis: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

PLOT - Program Lines of Trace - the source code lines behind trace messages

Examples: What a plot do we have here! The struggle against the monster database component and endless voyages across space boundaries.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

Today we introduce an icon for Stack Overflow (user mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Stack Overflow (kernel mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Modern pattern-driven software trace analysis on Microsoft and Citrix platforms urgently requires a practical guide and OpenTask plans to publish this summer the following book in both Practical Foundations and Systematic Software Fault Analysis series:

• Title: Citrix Common Diagnostic Facility (CDF) and Microsoft Event Tracing for Windows (ETW) Software Trace Analysis: Practical Foundations
• Author: Dmitry Vostokov
• Publisher: Opentask (August 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN: 1906717176
• ISBN-13: 978-1906717179
• Paperback: 200 pages

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

c’t – Magazin für Computertechnik has published a review of First Fault Software Problem Solving book: 

http://www.heise.de/ct/inhalt/2010/08/192/ (in German)

Fabian Röken kindly translated it into English:

No single large software package comes without errors. It seems that customers simply accept this, patiently waiting and hoping for patches or updates. Skwire sticks up for a more target-aimed approach: one will never get a faultless software, but it would already be a great improvement if flaws were already solved on their first occurrence (”first fault”) and not only after a long analysis (”second fault”).

The advantages are actually obvious. However, a corresponding stringent system architecture, as common on mainframes such as IBM’s z/OS, did not become prevalent in the PC market.

Skwire outlines the types of errors and strategies to resolve them in all details. His 40 years of experience, such as at IBM, shimmers through again and again. He puts emphasis on making sure that the reader understands the terminology he is using: “What is a problem in the first place?”, “What is a service point?” - in some cases he also explains specific metrics such as the “serviceability rating”.

His tool classification includes teaching tips, e.g. regarding the structure of a protocol in case of errors; or for tracking the important information how often an error must occur before a solution has to be approached. His suggestions equally address developers, designers, testers, managers - and the end user. In his last chapter he presents and reviews commercial tools in the first fault and second fault environment.

Skwire addresses a topic which is unfortunately very much neglected, and this alone already makes it worth enough to take a look at his book (***). Short quotations and humorous drawings relax the technical topic. If you are looking for an overview then you will be fine with this book. However, if you are a software developer looking for source code samples then you will search in vain. Skwire has released the book under the print-on-demand process. You will find it on Amazon, for example.

(Tobias Engler/fm)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It is time to start being systematic. In addition to all-encompassing multi-volume Memory Dump Analysis Anthology OpenTask starts Systematic Software Fault Analysis series with Crash Dump Analysis: Practical Foundations as the first book. It introduces basic definitions, tools, memory dump collection and preliminary analysis methods for Windows platforms including legacy versions. This practical reference guide is a must have for system administrators of Windows server platforms and client workstations, technical support engineers and general Windows users. It builds foundation for the second book Crash Dump Analysis for System Administrators and Support Engineers and the remaining tetralogy books Windows Crash Dump Analysis and Advanced Windows Crash Dump Analysis.

Product information:

• Title: Crash Dump Analysis: Practical Foundations (Windows Edition, Systematic Software Fault Analysis Series)
• Authors: Dmitry Vostokov
• Publisher: Opentask (May 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-98-8
• Paperback: 100 pages

Fromt cover:

Table of Contents to be published soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The book is about to be submitted for publication. It has 68 full color illustrations. Here is TOC:

Table of Contents

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Module Variety pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Anthology lacks full color inserts. This is rather a current limitation of POD technology that OpenTask publisher uses at the present time. The solution previously announced a year ago is to print a separate full color title with selected articles and illustrations. Finally, it is about to be published next month. The book front and back covers are collages from covers of individual memory dump analysis volumes:

TOC will be available soon. Here is the book data:

• Title: Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
• Author: Dmitry Vostokov
• Publisher: OpenTask (May 2010)
• Language: English
• Product Dimensions: 21.6 x 14.0
• Paperback: 110 pages
• ISBN-13: 978-1906717698

OpenTask also plans a separate color supplement for volumes 4 and 5 once they are published. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -