Software Diagnostics Library

Followed by my search for a periodic table of software defects I came up with a novel approach called Generative Debugging partially borrowed from principles and parameters generative linguistics framework. For debugging and post-construction software problem solving purposes we have Behavioral Principles and Structural Parameters. The parameters allow us to describe existing and even generate new software defects (I’m also considering Structural Principles and Behavioural Parameters but whether this is fruitful needs to be explored and seen. We may also consider the name “generative debugging” as a “naming” counterpart to software construction problem solving called generative programming. So the former could also be used to generate problem solutions faster in automatic fashion. Both, in the future, may be combined into a unified generative software problem solving.

As a first example, consider an application functional hang with a thread blocked by a pending I/O request packet (IRP). It is composed from general behavioral principle of blocked (or unresponsive) activity and structural parameters including Thread and IRP.

In the following parts many software incidents will be explored from this generative perspective.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m to present pattern-driven software trace analysis with examples from application and desktop delivery environments featuring Memory Dump Analysis Services at the forthcoming E2E Virtualization Conference (PubForum) in Dublin on 13th of May, 2011. Topics include a case study covering simultaneous analysis of software traces and memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services plans a free Webinar in July to introduce a uniform methodology and tools for analysis of memory in cyber intelligence and counterintelligence activities.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Adjoint Thread invariants like PID can be reused giving rise to curious CDF (ETW) traces where two separate execution entities are glued together in one trace. For example, in one trace we see AppA and AppB sharing the same PID

#      Module  PID  TID   Time         Message
[…]
242583 ProcMon 5492 9476  11:04:33.398 LoadImageEvent for ImageName: …\AppA.exe PID: 5492
256222 ProcMon 5492 9476  11:04:50.036 ProcessDestroyEvent for PPID: 12168 PID: 5492 
274887 ProcMon 5492 10288 11:05:18.235 LoadImageEvent for ImageName: …\AppB.exe PID: 5492 
[…]

Other similar examples might include different instances of components sharing the same name, source code or even, in general, periodic tracing sessions appended to the end of the same trace file although we are thinking that the latter should be a separate pattern. We named this pattern Glued Activity by an analogy of different thread strings glued together (in general, manifiolds glued along their boundaries). Another name might be along the line of adjoint thread ID reuse (ATID Reuse).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first Webinar to start an in-depth discussion of a uniform methodology and tools for analysis of crashes, hangs, and other types of abnormal software behaviour in cloud environments.

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free Webinar

Date: 13th of June 2011
Time: 18:45 (BST) 13:45 (EST) 10:45 (PST)
Duration: 60 minutes

Topics include:

• What’s old
• What’s new
• Orbifold memory spaces
• Methodology and best practices
• Cloud-specific patterns
• Hands-on exercises

Prerequisites: working knowledge of basic memory dump analysis or live debugging.

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/597101206

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally, after the numerous delays, the first Windows Debugging Notebook is almost ready for publication by the end of this month with a new cover and a foreword written by Mario Hewardt, the author of Advanced Windows Debugging and Advanced .NET Debugging books:

It is the first notebook because we have decided to split it into 3 different memory space volumes and one additional overview volume for concepts and tools:

- Windows Debugging Notebook: Essential User Space WinDbg Commands (ISBN: 978-1906717001 and 978-0955832857)
- Windows Debugging Notebook: Essential Kernel Space WinDbg Commands (ISBN: 978-1908043146)
- Windows Debugging Notebook: Essential Managed Space WinDbg .NET Commands (ISBN: 978-1908043153)
- Windows Debugging Notebook: Essential Concepts and Tools (ISBN: 978-1908043160)

On a bookshelf you would be able to distinguish between the first 3 volumes by a 3 color spine stripe with an appropriate check on it: red for kernel, blue for user, and green for managed space.

The Table of Contents has been changed too.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a kernel space counterpart to Custom Exception Handler pattern in user space. In the following stack trace below we see that DriverA code intercepted an access violation exception resulted from dereferencing a NULL pointer and generated a custom bugcheck:

kd> !analyze -v

[...]

EXCEPTION_RECORD: fffff8801c757158 -- (.exr 0xfffff8801c757158)
ExceptionAddress: fffff88003977de1 (DriverA!foo+0x0000000000000381)
ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
  Parameter[0]: 0000000000000000
  Parameter[1]: 0000000000000070
Attempt to read from address 0000000000000070

TRAP_FRAME: fffff8801c757200 -- (.trap 0xfffff8801c757200)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff8a00da3f3c0
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff88003977de1 rsp=fffff8801c757390 rbp=fffffa8009a853f0
 r8=0000000000000000 r9=0000000000000000 r10=006800740020006e
r11=fffff8a00da3f3c6 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
DriverA!foo+0×381:
fffff880`03977de1 0fb74070 movzx eax,word ptr [rax+70h] ds:0703:0070=????
Resetting default scope

[...]

kd> kL 100
Child-SP          RetAddr           Call Site
fffff880`1c7560f8 fffff880`039498f7 nt!KeBugCheckEx
fffff880`1c756100 fffff880`039352a0 DriverA!MyBugCheckEx+0×93
fffff880`1c756140 fffff800`016f1d1c DriverA!MyExceptionFilter+0×1d0
fffff880`1c756210 fffff800`016e940d nt!_C_specific_handler+0×8c
fffff880`1c756280 fffff800`016f0a90 nt!RtlpExecuteHandlerForException+0xd
fffff880`1c7562b0 fffff800`016fd9ef nt!RtlDispatchException+0×410
fffff880`1c756990 fffff800`016c2d82 nt!KiDispatchException+0×16f
fffff880`1c757020 fffff800`016c18fa nt!KiExceptionDispatch+0xc2
fffff880`1c757200 fffff880`03977de1 nt!KiPageFault+0×23a
fffff880`1c757390 fffff880`03977754 DriverA!foo+0×381
fffff880`1c757430 fffff880`0396f006 DriverA!bar+0×74
[…]
fffff880`1c7579b0 fffff800`019a6e0a DriverA!QueryInformation+0×30b
fffff880`1c757a70 fffff800`016c2993 nt!NtQueryInformationFile+0×535
fffff880`1c757bb0 00000000`76e5fe6a nt!KiSystemServiceCopyEnd+0×13
00000000`0a08dfe8 00000000`00000000 0×76e5fe6a

kd> !exchain
24 stack frames, scanning for handlers...
Frame 0×05: nt!RtlpExecuteHandlerForException+0xd (fffff800`016e940d)
  ehandler nt!RtlpExceptionHandler (fffff800`016e93d0)
Frame 0×07: nt!KiDispatchException+0×16f (fffff800`016fd9ef)
  ehandler nt!_GSHandlerCheck_SEH (fffff800`0169aec0)
Frame 0×0b: DriverA!bar+0×74 (fffff880`03977754)
  ehandler DriverA!__GSHandlerCheck (fffff880`039a12fc)
[…]
Frame 0×14: DriverA!QueryInformation+0×30b (fffff880`039303ab)
  ehandler DriverA!_C_specific_handler (fffff880`039a1864)
Frame 0×15: nt!NtQueryInformationFile+0×535 (fffff800`019a6e0a)
  ehandler nt!_C_specific_handler (fffff800`016f1c90)
Frame 0×16: nt!KiSystemServiceCopyEnd+0×13 (fffff800`016c2993)
  ehandler nt!KiSystemServiceHandler (fffff800`016c2580)
Frame 0×17: error getting module for 0000000076e5fe6a

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Just curious about wetware patterns. I think that if any meaningful patterns are found in wetware they would be on a micro scale…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Data Correlation is a general pattern where values found in different parts of a memory dump correlate between each other according to some rules, for example, in some proportion. Here we show a variant for function parameters.

A process user memory dump showed a C++ exception:

0:000> kL
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
0012e950 78158e89 kernel32!RaiseException+0x53
0012e988 7830770c msvcr80!_CxxThrowException+0×46
0012e99c 783095bc mfc80u!AfxThrowMemoryException+0×19
0012e9b4 02afa8ca mfc80u!operator new+0×27
0012e9c8 02b0992f ModuleA!std::_Allocate<…>+0×1a
0012e9e0 02b09e7c ModuleA!std::vector<double,std::allocator<double> >::vector<double,std::allocator<double> >+0×3f
[…]

We suspected an out-of-memory condition and looked for function parameters:

0:000> kv 5
ChildEBP RetAddr  Args to Child             
0012e950 78158e89 e06d7363 00000001 00000003 kernel32!RaiseException+0x53
0012e988 7830770c 0012e998 783b0110 783c8d68 msvcr80!_CxxThrowException+0x46
0012e99c 783095bc 0000a7c0 0012ea40 000014f8 mfc80u!AfxThrowMemoryException+0x19
0012e9b4 02afa8ca 0000a7c0 089321b0 089321f0 mfc80u!operator new+0×27 (FPO: [Uses EBP] [1,0,0])
0012e9c8 02b0992f 000014f8 00000000 00000008 ModuleA!std::_Allocate<…>+0×1a (FPO: [2,3,0])

Because of FPO optimization we originally thought that stack arguments would be invalid. However, bearing in mind the function prototype and semantics of operator new and std::vector double element type we immediately see the correlation between 0xa7c0 and 0×14f8 which are proportional to sizeof(double) == 8:

0:000> ? 0000a7c0/000014f8
Evaluate expression: 8 = 00000000`00000008

We therefore conclude without looking at disassembly that memory allocation size was 42944 bytes:

0:000> .formats 0000a7c0
Evaluate expression:
  Hex:     00000000`0000a7c0
  Decimal: 42944
  Octal:   0000000000000000123700
  Binary:  00000000 00000000 00000000 00000000 00000000 00000000 10100111 11000000
  Chars:   ……..
  Time:    Thu Jan 01 11:55:44 1970
  Float:   low 6.01774e-041 high 0
  Double:  2.12172e-319

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m speechless for this occasion, look with awe at what has been done, and can only show a slide from the yesterday’s presentation Introduction to Pattern-Driven Software Problem Solving:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The presentation materials from the webinar (25th of March, 2011) are available for download:

http://www.dumpanalysis.com/PDSPSI-materials

Thanks to everyone who registered and attended!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services opens Debugging Bureau to complement crash and hang memory dump analysis audit:

http://www.dumpanalysis.com/debugging-bureau

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Stack Overflow (software implementation) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

While working on Pattern-Driven Software Problem Solving Webinars I needed a reference page that lists all Memory Dump Analysis Anthology volumes:

Advanced Software Debugging Reference

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to my Mod N Reading System I had to suspend this blog because N became large (about 100 or more books) and there wasn’t enough read material to review. Now I accumulated some books fully read from cover to cover and many others read to a sufficient number of pages to make an opinion and suggest recommendations. So I’m resuming this blog and plan to write about the next book tomorrow. It will be a history of social sciences. I plan to introduce a new book every day or so. I also made an alteration to Mod N reading to include thematic encyclopedias. Stay tuned.

- Dmitry Vostokov @ LiterateScientist.com -

Sometimes we see Truncated Stack Trace with missing stack frames. For example, in one incident, after enabling user mode stack trace database for a memory leaking application we got these entries from the growing heap segment (other segments had non-truncated saved stack traces):

0bdc1350: 40010 . 40010 [101] - busy (3fff8) Internal

    7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
    77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
    77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
    7702661f: ntdll!RtlAllocateHeap+0x0000017c

0be01360: 40010 . 40010 [101] - busy (3fff8) Internal

    7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
    77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
    77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
    7702661f: ntdll!RtlAllocateHeap+0x0000017c

0be41370: 40010 . 40010 [101] - busy (3fff8) Internal

    7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
    77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
    77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
    7702661f: ntdll!RtlAllocateHeap+0x0000017c

Truncated traces are different from incorrect stack traces because their surviving part is correct. How can we find the rest of such stack traces? Here we can suggest to look at other heap segments and see allocations of the same size. If a truncated trace comes from a stack trace collection we can compare it with a non-truncated thread stack from another process instance having the same thread position. Other suggestions are welcome too, especially, for the mysterious memory leak shown above.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The forthcoming Software Behavior Analysis Patterns and Checklists: Corporate Reference Poster Edition for Windows Crash and Hang Memory Dump and Software Trace Analysis (ISBN: 978-1908043030) will feature Analysis Compass based on the reversal of an extended version of pattern-to-command table published in the printed version of Debugged! MZ/PE March, 2009 issue. Here’s a sneak preview screenshot of this feature:

The notation F”Function” means searching for frames having the given Function name in stack traces from Stack Trace Collection. S”String” means searching for String in the output of WinDbg commands, for example, !process 0 ff.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Frequently caught myself recognizing bit values like 100084 when I’m looking at car plate numbers. Sometimes I see a number and wonder whether it is a valid window handle.

Do you also have similar perceptions? Please let me know if you have different memceptions

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

“If you want to be thought a” debugger “, always” talk about bugs.

Logan Pearsall Smith

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

“… he who has not been in” support “does not know what” debugging “is.”

Attributed to Leo Tolstoy, by Adam Ulam, The Bolsheviks

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -