Software Diagnostics Library

A page to reference all different kinds of exception related patterns is necessary, so I created this post:

• Multiple Exceptions (user mode)
• Multiple Exceptions (kernel mode)
• Multiple Exceptions (managed space)
• Multiple Exceptions (stowed)
• Hidden Exception (user space)
• Hidden Exception (kernel space)
• Hidden Exception (managed space)
• Managed Code Exception
• Custom Exception Handler (user space)
• Custom Exception Handler (kernel space)
• Nested Exceptions (unmanaged code)
• Nested Exceptions (managed code)
• C++ Exception
• Mixed Exception
• Exception Stack Trace
• Invalid Exception Information
• Problem Exception Handler
• Handled Exception (user space)
• Handled Exception (.NET CLR)
• Handled Exception (kernel space)
• FPU Exception
• Translated Exception
• Stored Exception
• Exception Module
• Software Exception
• Distributed Exception (managed code)
• Exception Reporting Thread
• COM Exception
• Saved Exception Context (.NET)
• Exception Collection
• Near Exception

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One pattern I missed is called FPU Exception and it sometimes happens where you least expect it. Here’s extract from one crash dump raw stack analysis showing exception context, record and the usage of r WinDbg command variant to display FPU registers:

0:002> dps 056c1000 057c0000  
[...]
057bdee0  00000008
057bdee4  00000000
057bdee8  057bed6c
057bdeec  0d6e3130
057bdef0  057c0000
057bdef4  057b9000
057bdef8  006e3138
057bdefc  057be200
057bdf00  7c90e48a ntdll!KiUserExceptionDispatcher+0xe
057bdf04  057bed6c
057bdf08  057bdf2c
057bdf0c  057bdf14
057bdf10  057bdf2c
057bdf14  c0000090
057bdf18  00000010
057bdf1c  00000000
057bdf20  79098cc0 mscorjit!Compiler::FlatFPIsSameAsFloat+0xd
057bdf24  00000001
057bdf28  00000000
057bdf2c  0001003f
057bdf30  00000000
057bdf34  00000000
057bdf38  00000000
057bdf3c  00000000
057bdf40  00000000
057bdf44  00000000
057bdf48  ffff1372
057bdf4c  fffffda1
057bdf50  ffffbfff 
[…]

0:002> .cxr 057bdf2c
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
mscorwks!SOTolerantBoundaryFilter+0x22:
79f5236b d9059823f579    fld     dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000

0:002> .exr 057bdf14
ExceptionAddress: 79098cc0 (mscorjit!Compiler::FlatFPIsSameAsFloat+0x0000000d)
   ExceptionCode: c0000090
  ExceptionFlags: 00000010
NumberParameters: 1
   Parameter[0]: 00000000

0:002> !error c0000090
Error code: (NTSTATUS) 0xc0000090 (3221225616) - {EXCEPTION}  Floating-point invalid operation.

0:002> rMF
Last set context:
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
fpcw=1372: rn 64 pu–d-  fpsw=FDA1: top=7 cc=1101 b-p—-i  fptw=BFFF
fopcode=045D  fpip=001b:79098cc0  fpdp=0023:057bea7c
st0=-1.#IND00000000000000000e+0000  st1= 0.006980626232475338220e-4916
st2= 6.543831490564206840810e-4932  st3=-0.003025663186207448300e+2614
st4= 2.000000000000000000000e+0000  st5= 6.291456000000000000000e+0006
st6= 1.000000000000000000000e+0000  st7= 2.500000000000000000000e-0001
mscorwks!SOTolerantBoundaryFilter+0×22:
79f5236b d9059823f579    fld     dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If memory is the foundation of everything and the first principle then what about matter? We view matter as curvature of memory (currently metaphorically) and also as a constraining “filter” device that limits (and processes) memories. The latter view of limits is similar to some theories viewing brain (body) as a constraining device for mental reality (consciousness)^*.

(*) Irreducible Mind (Kelly & Kelly et al.), pp. 28-29

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Frequently we observe that several trace messages form a single semantic unit we call Macrofunction where individual trace messages serve the role of microfunctions. We borrowed this idea and distinction from functionalist linguistics. An example would be a software trace fragment where messages log an attempt to update a database:

#     Module   PID  TID   Time         Message
[...]
42582 DBClient 5492 9476  11:04:33.398 Opening connection
[...]
42585 DBClient 5492 9476  11:04:33.398 Sending SQL command
[...]
42589 DBServer 6480 10288 11:04:33.399 Executing SQL command
[...]
42592 DBClient 5492 9476  11:04:33.400 Closing connection
[...]

Please note that these macrofunctions need not be from the same ATID in the traditional sense like in the example above unless we form adjoint threads from certain fragments like “DB”.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

LoL - Law of Large

Examples: Q. How did you resolve this support case? A. LoL number!

Explanation: The more support incidents you get, the larger their tracking numbers. So at some stage the law of large numbers comes into effect: there is always a similar incident in the past. Don’t confuse with LOL.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a new exiting book project I’m working on now scheduled for release in 2012 with ISBN 978-1908043337. If your company would like to have its programs considered for inclusion please let me know and send a copy just in case I would need to include screenshots. I’ll update about this project soon.

I was recently asked to provide explanation on how to analyze (A)LPC wait chains and the output of !lpc and !alpc commands in kernel and complete memory dumps and so I compiled these patterns:

• LPC Deadlock
• LPC Wait Chain
• Paged Out Data
• Process Object Wait Chain (+ ALPC)
• Blocked Queue (ALPC)
• Semantic Structures (ALPC)
• RPC Wait Chain (+ how to distinguish between LPC and RPC)
• Screwbolt Wait Chain

LPC case studies:

http://www.dumpanalysis.org/blog/index.php/2008/12/19/blocked-gui-thread-wait-chain-and-virtualized-process-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2008/12/24/insufficient-memory-handle-leak-wait-chain-deadlock-inconsistent-dump-and-overaged-system-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/03/11/coupled-processes-wait-chains-message-box-waiting-thread-time-paged-out-data-incorrect-stack-trace-hidden-exception-unknown-component-and-execution-residue-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/06/09/inconsistent-dump-blocked-threads-wait-chains-incorrect-stack-trace-and-process-factory-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/08/11/stack-trace-collection-suspended-threads-not-my-version-special-process-main-thread-and-blocked-lpc-chain-threads-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/11/20/stack-trace-collection-missing-threads-waiting-time-critical-section-and-lpc-wait-chains-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/02/27/inconsistent-dump-stack-trace-collection-lpc-thread-process-executive-resource-wait-chains-missing-threads-and-waiting-thread-time-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/07/18/stack-trace-collection-special-process-lpc-and-critical-section-wait-chains-blocked-thread-coupled-machines-thread-waiting-time-and-irp-distribution-anomaly-pattern-cooperation/

ALPC case studies:

http://www.dumpanalysis.org/blog/index.php/2009/09/18/alpc-wait-chain-missing-threads-message-box-zombie-and-special-processes-pattern-cooperation/

RPC target:

http://www.dumpanalysis.org/blog/index.php/2008/07/11/in-search-of-lost-pid/

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

During the previous several months some companies and individuals expressed their interest in the training (the next one is scheduled for January) but its time was not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

• Title: Accelerated .NET Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
• Description: The full transcript of Memory Dump Analysis Services Training with 7 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers 20 .NET memory dump analysis patterns plus additional unmanaged patterns.
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Publisher: OpenTask (November 2011)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 204 pages
• ISBN-13: 978-1908043320

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a name for a virtual country where I’m a virtual citizen. This coined word is the latest addition to Citrixware and Citrixofication together with I Love Citrix social media logo.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a second long-term initiative for 2012 to design and develop memory-oriented operating system where memory is the foundation of the whole architecture from the ground up. More on this later as the announcement date and time memory pattern 11-11-11 11:11 is quickly approaching

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the new initiatives for 2012 is the development of SPDL (Software Problem Description Language). Its purpose is automatic generation of a software troubleshooting tool(s) based on the description of a problem. Here software problem means a post-construction problem as outlined in Introduction to Pattern-Driven Software Problem Solving. The tool construction will utilize an expanded set of DebugWare and Workaround patterns together with the refind version of RADII software development process. This will also provide necessary effectiveness, efficiency and enhanced problem solving capabilities to existing TaaS (Tools as a Service) implementations that are limited in the number of tools they offer.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Hidden Parameter pattern is a variant of Execution Residue and String Parameter where we have parameters left out from stack trace due to register calling conventions and compiler optimizations. However, raw stack analysis in a region around stack frames of interest we find what we are looking for. Here’s an example from an x64 system blocked thread waiting for data from a named pipe:

0: kd> kL
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
fffffa60`2c3627d0 fffff800`018b90fa nt!KiSwapContext+0x7f
fffffa60`2c362910 fffff800`018add3b nt!KiSwapThread+0x13a
fffffa60`2c362980 fffff800`01b2121f nt!KeWaitForSingleObject+0x2cb
fffffa60`2c362a10 fffff800`01b319b6 nt!IopXxxControlFile+0xdeb
fffffa60`2c362b40 fffff800`018b68f3 nt!NtFsControlFile+0x56
fffffa60`2c362bb0 00000000`778d6eaa nt!KiSystemServiceCopyEnd+0x13
00000000`11f4da68 00000000`77767b6e ntdll!ZwFsControlFile+0xa
00000000`11f4da70 000007fe`ff94abc8 kernel32!WaitNamedPipeW+0×22f
00000000`11f4db60 000007fe`ff98a32d RPCRT4!NdrProxyForwardingFunction255+0×814d
00000000`11f4dc30 000007fe`ff98918b RPCRT4!OSF_CCONNECTION::TransOpen+0xcd
00000000`11f4dcc0 000007fe`ff988f9b RPCRT4!OSF_CCONNECTION::OpenConnectionAndBind+0×17b
00000000`11f4dd90 000007fe`ff988ec6 RPCRT4!OSF_CCALL::BindToServer+0xbb
00000000`11f4de40 000007fe`ff983368 RPCRT4!OSF_BINDING_HANDLE::InitCCallWithAssociation+0xa5
00000000`11f4dea0 000007fe`ff983220 RPCRT4!OSF_BINDING_HANDLE::AllocateCCall+0×118
00000000`11f4dfd0 000007fe`ffa1f740 RPCRT4!OSF_BINDING_HANDLE::NegotiateTransferSyntax+0×30
00000000`11f4e020 000007fe`ffa1fecb RPCRT4!Ndr64pClientSetupTransferSyntax+0×200
00000000`11f4e080 000007fe`ffa20281 RPCRT4!NdrpClientCall3+0×6b
00000000`11f4e2d0 000007fe`fe087c8c RPCRT4!NdrClientCall3+0xdd
[…]

Even if we disassemble the return address of a caller of WaitNamedPipeW function we won’t easily find the passed first string parameter (named pipe name) unless we do a substancial reverse engineering and data flow analysis:

0: kd> ub 000007fe`ff94abc8
RPCRT4!_imp_load_getaddrinfo+0×7:
000007fe`ff94ab9f jmp     RPCRT4!_tailMerge_WS2_32_dll (000007fe`ff94cef8)
000007fe`ff94aba4 call    qword ptr [RPCRT4!_imp_GetLastError (000007fe`ffa2d528)]
000007fe`ff94abaa mov     r12d,eax
000007fe`ff94abad cmp     r12d,0E7h
000007fe`ff94abb4 jne     RPCRT4!NdrProxyForwardingFunction255+0×8193 (000007fe`ff99c8fb)
000007fe`ff94abba mov     edx,3E8h
000007fe`ff94abbf mov     rcx,rsi
000007fe`ff94abc2 call    qword ptr [RPCRT4!_imp_WaitNamedPipeW (000007fe`ffa2d3f8)]

So dumping raw stack date around corresponding frames give us pipe name clue and possible service to look further:

0: kd> dpu 00000000`11f4da70
00000000`11f4da70  00000000`11f4dba8 “\\.\PIPE\ServiceArpc”
00000000`11f4da78  00000000`00000000
00000000`11f4da80  00000000`00000000
00000000`11f4da88  00000000`000003e8
00000000`11f4da90  00000000`11f4db30
00000000`11f4da98  00000000`00110018
00000000`11f4daa0  00000000`0d9001a0
00000000`11f4daa8  00000000`0000001a
00000000`11f4dab0  00000000`00000000
00000000`11f4dab8  00000000`00000000
00000000`11f4dac0  00000000`0020000c
00000000`11f4dac8  00000000`0d9001e2 “ServiceArpc”
00000000`11f4dad0  00000000`00000000
00000000`11f4dad8  00000000`00000000
00000000`11f4dae0  00000000`00240022

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was recently asked to provide explanation on how to analyze the output of !locks command in kernel and complete memory dumps and I compiled these patterns:

• _ERESOURCE Deadlock
• Mixed Object Deadlock
• Self Deadlock
• High Contention
• Accidental Lock
• _ERESOURCE Wait Chain
• Swarm of Shared Locks
• Semantic Split
• Livelock

and case studies:

http://www.dumpanalysis.org/blog/index.php/2008/07/17/multiple-patterns-in-a-dump-case-study/

http://www.dumpanalysis.org/blog/index.php/2008/10/29/10-common-mistakes-in-memory-analysis-part-3/

http://www.dumpanalysis.org/blog/index.php/2008/12/12/wait-chain-and-spiking-thread-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/02/16/insufficient-memory-handle-leak-process-factory-high-contention-and-busy-system-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/03/03/busy-system-blocked-threads-wait-chains-and-deadlock-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2009/12/17/wait-chain-blocked-thread-waiting-thread-time-irp-distribution-anomaly-and-stack-trace-collection-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/02/27/inconsistent-dump-stack-trace-collection-lpc-thread-process-executive-resource-wait-chains-missing-threads-and-waiting-thread-time-pattern-cooperation/

http://www.dumpanalysis.org/blog/index.php/2010/06/03/succession-of-patterns-part-2/

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The slides are available from Memory Dump Analysis Services:

http://www.dumpanalysis.com/Training/Accelerated-NET-Memory-Dump-Analysis-Public.pdf

It also offers the new training sessions in January, 2012:

http://www.dumpanalysis.com/accelerated-net-memory-dump-analysis

http://www.dumpanalysis.com/accelerated-windows-memory-dump-analysis

Also the registration for Advanced training session in December is still open:

http://www.dumpanalysis.com/advanced-windows-memory-dump-analysis

There are also new coming courses for 2012 so stay tuned!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:

• 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
• 4 new pattern interaction case studies
• 11 new trace analysis patterns
• New Debugware pattern
• Introduction to UI problem analysis patterns
• Introduction to intelligence analysis patterns
• Introduction to unified debugging pattern language
• Introduction to generative debugging, metadefect template library and DNA of software behaviour
• The new school of debugging and trends
• .NET memory dump analysis checklist
• Software trace analysis checklist
• Introduction to close and deconstructive readings of a software trace
• Memory dump analysis compass
• Computical and Stack Trace Art
• The abductive reasoning of Philip Marlowe
• Orbifold memory space and cloud computing
• Memory worldview
• Interpretation of cyberspace
• Relationship of memory dumps to religion
• Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5

Product information:

• Title: Memory Dump Analysis Anthology, Volume 6
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 300 pages
• Publisher: Opentask (December 2011)
• ISBN-13: 978-1-908043-19-1

• Hardcover: 300 pages
• Publisher: Opentask (January 2012)
• ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have Linked Messages through some common parameter or attribute. One such example can be found in ETW traces related to kernel process creation notifications. Here we got adjoint thread for module PIDNotify:

#      Module    PID  TID  Time         Message
[...]
128762 PIDNotify 1260 6208 15:53:15.691 Create: ParentID 0x000004EC PID 0×000018D4
[…]
128785 PIDNotify 6356 6388 15:53:15.693 Load: ImageName \Device\HarddiskVolume1\Windows\System32\abscript.exe PID 0×000018D4
[…]
131137 PIDNotify 6356 4568 15:53:15.936 Create: ParentID 0×000018D4 PID 0×00001888
[…]
131239 PIDNotify 6280 6376 15:53:15.958 Load: ImageName \Device\HarddiskVolume1\Windows\System32\wscript.exe PID 0×00001888
[…]
132899 PIDNotify 6356 5704 15:53:16.462 Create: ParentID 0×000018D4 PID 0×00001FD0
[…]
132906 PIDNotify 8144 7900 15:53:16.464 Load: ImageName \Device\HarddiskVolume1\Windows\System32\cmd.exe PID 0×00001FD0
[…]

We see that messages 128762 and 128785 are linked through PID parameter and linked to messages 131137 and 132899 through PID - ParentID parameter relationship. Similar linkages exist for messages 131137 / 131239 and 132899 / 132906.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -