Encyclopedia of Crash Dump Analysis Patterns, Third Edition

September 19th, 2020

Available with a 50% discount (for 40 USD) until the 23th of September 2020:

https://www.patterndiagnostics.com/encyclopedia-crash-dump-analysis-patterns

Memory Dump Analysis Anthology 13-Volume Set

September 19th, 2020

The content of this site is available in edited and revised PDF format with a significant discount for volume set:

https://www.patterndiagnostics.com/mdaa-volumes

Alternatively, analysis pattern and theoretical content is included in Pattern-Oriented Software Diagnostics and Anomaly Detection Reference Set:

https://www.patterndiagnostics.com/pattern-oriented-software-diagnostics-reference-set

Trace Analysis Patterns (Part 201)

September 19th, 2020

If messages from (Adjoint) Thread of Activity also have associated traces (Fiber Bundle) then the latter messages data, for example, module names, can be interlinked with corresponding Adjoint Threads of Activity, thus forming “two-dimensional” Weave of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 272)

September 18th, 2020

We introduced Procedure Call Chain in Stack Trace Collection and Coupled Processes analysis patterns. It is a stack trace formed by gluing stack trace fragments (for example, from LPC/ALPC and RPC Wait Chains) after removing middleware subtraces. For example, we use the following case study (32-bit for less space), follow its Wait Chain, and construct Procedure Call Chain (we use different colors to show gluing):

b88dccb8 804e1bf2 nt!KiSwapContext+0×2f
b88dccc4 804e1c3e nt!KiSwapThread+0×8a
b88dccec 8056dff6 nt!KeWaitForSingleObject+0×1c2
b88dcd50 804dd99f nt!NtWaitForSingleObject+0×9a
b88dcd50 7c90e514 nt!KiFastCallEntry+0xfc (TrapFrame @ b88dcd64)
036ef714 7c90df5a ntdll!KiFastSystemCallRet
036ef718 7c91b24b ntdll!ZwWaitForSingleObject+0xc
036ef7a0 7c901046 ntdll!RtlpWaitForCriticalSection+0×132
036ef7a8 6648a33b ntdll!RtlEnterCriticalSection+0×46
036ef7b0 6648c2ed ipnathlp!FwLock+0xa
036ef808 6648c705 ipnathlp!FwDynPortAdd+0×1d
036ef8c4 77e799f4 ipnathlp!FwOpenDynamicFwPort+0×125
00a9ef9c 662dafa9 hnetcfg!FwOpenDynamicFwPort+0×1b
00a9f048 71a55025 hnetcfg!IcfOpenDynamicFwPort+0×6a
00a9f0e4 71a590f2 mswsock!WSPBind+0×332
00a9f200 71ab2fd7 mswsock!WSPSendTo+0×230
00a9f250 76f252c0 WS2_32!sendto+0×88
00a9f280 76f251ea DNSAPI!SendMessagePrivate+0×18d
00a9f2a0 76f2517c DNSAPI!SendUsingServerInfo+0×33
00a9f2c8 76f25436 DNSAPI!SendUdpToNextDnsServers+0×80
00a9f314 76f24dec DNSAPI!Dns_SendAndRecvUdp+0×121
00a9f34c 76f24d20 DNSAPI!Dns_SendAndRecv+0×7b
00a9f37c 76f24a7d DNSAPI!Query_SingleName+0×8b
00a9f3b0 7677373a DNSAPI!Query_Main+0×11a
00a9f3c8 7677303f dnsrslvr!ResolverQuery+0×48
00a9f8bc 77e799f4 dnsrslvr!R_ResolverQuery+0×111
00a8f4c4 76f2357b DNSAPI!R_ResolverQuery+0×1b
00a8f520 71a526c6 DNSAPI!DnsQuery_W+0×14f
00a8f554 71a5266f mswsock!HostentBlob_Query+0×29
00a8f580 71a51b0a mswsock!Rnr_DoDnsLookup+0×7d
00a8f9c8 71ab32b0 mswsock!NSPLookupServiceNext+0×533
00a8f9e0 71ab3290 WS2_32!NSPROVIDER::NSPLookupServiceNext+0×17
00a8f9fc 71ab325a WS2_32!NSPROVIDERSTATE::LookupServiceNext+0×1c
00a8fa28 71ab31f8 WS2_32!NSQUERY::LookupServiceNext+0xae
00a8fa48 76f775eb WS2_32!WSALookupServiceNextW+0×78
00a8faec 76f6a9d2 WLDAP32!GetHostByNameW+0xef
00a8fb38 76f6667b WLDAP32!OpenLdapServer+0×435
00a8fb58 76f6fb05 WLDAP32!LdapConnect+0×169
00a8fef8 76f704f3 WLDAP32!LdapBind+0×34
00a8ff20 5e95651a WLDAP32!ldap_bind_sW+0×2c
00a8ff68 5e95a887 PAUTOENR!AERobustLdapBind+0xc9
00a8ffb4 7c80b729 PAUTOENR!AEMainThreadProc+0xef
00a8ffec 00000000 kernel32!BaseThreadStart+0×37

This similar to Glued Stack Trace which is produced from fragments that belong to one stack region.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 200)

September 13th, 2020

Trace and log analysis patterns may be additionally applied not only to a database like tables but also to texts (as an example of general trace and log analysis). Sentences may form trace messages with paragraphs and chapters corresponding to traditional ATIDs (IDs for Adjoint Threads of Activity) such as TID and PID in the most simple syntax mapping case, and certain sentences may be interpreted as Silent Messages.

Different attribute generation schemas may be used, for example, selected vocabulary may be used to assign TID numbers. More complex cases may require paratexts, supplementary texts providing additional structure and semantic information like in the case of Paratext memory analysis pattern, the case of extended traces.

The opposite process of converting traces and logs to text is also possible with additional paratext generation if necessary. We call this two-way analysis pattern Text Trace. After converting texts to logs it is possible to apply the majority of trace and log analysis patterns from the catalog.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 199)

September 13th, 2020

Several Strands of Activity from different types of ATIDs (Adjoint Threads of Activity) combine into Cord of Activity:

Between cord and rope analogies we chose cord as having “ord” (ordinal) in it (and c as cardinal). It is also possible to combine several Cords of Activity from different traces (Trace Dimension) to form a “cable-laid rope”. We don’t introduce a separate pattern here since in the resulting Trace Mask we have new Cord of Activity due to the additionally created ATID type referencing former separate traces and logs. Data references in messages may provide additional braiding via Braids of Activity.

We started with strands (we got the idea from the discussion of ethnomathematics where strand analysis was mentioned) but then we found the following useful discussion on rope terminology: “Art and Science of Rope“.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 198)

September 12th, 2020

Strand of Activity combines different Threads of Activity or Adjoint Threads of Activity of the same type.

Strands extend cable and rope composition metaphors that start with Fibers of Activity, and continue with threads and Braids of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Frame Patterns

August 29th, 2020

A page to reference all different kinds of stack trace frames is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 271)

August 29th, 2020

Often a debugger is not able to reconstruct a stack trace correctly, for example, when symbols to guide the process are not available due to Reduced Symbol Information or complete absence due to Unloaded Module:

0:008> k
# ChildEBP RetAddr
00 0250f4b8 76d21775 ntdll!NtWaitForMultipleObjects+0x15
01 0250f554 75c419fc KERNELBASE!WaitForMultipleObjectsEx+0x100
02 0250f59c 75c4268c kernel32!WaitForMultipleObjectsExImplementation+0xe0
03 0250f5b8 75c681fc kernel32!WaitForMultipleObjects+0x18
04 0250f624 75c680bb kernel32!WerpReportFaultInternal+0x186
05 0250f638 75c679b0 kernel32!WerpReportFault+0x70
06 0250f648 75c6792f kernel32!BasepReportFault+0x20
07 0250f6d4 00e21e86 kernel32!UnhandledExceptionFilter+0x1af
08 0250f6f0 75c803cf ModuleA!UnhandledExceptionFilter+0x3d
09 0250f778 77e250d7 kernel32!UnhandledExceptionFilter+0x127
0a 0250f780 77e24fb4 ntdll!__RtlUserThreadStart+0x62
0b 0250f794 77e24e59 ntdll!_EH4_CallFilterFunc+0x12
0c 0250f7bc 77e134a1 ntdll!_except_handler4+0x8e
0d 0250f7e0 77e13473 ntdll!ExecuteHandler2+0x26
0e 0250f804 77e13414 ntdll!ExecuteHandler+0x24
0f 0250f890 77dc0133 ntdll!RtlDispatchException+0x127
10 0250f890 68a8e0ca ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
11 0250fd58 02c45f58 <Unloaded_ModuleB.dll>+0x1e0ca
12 0250fd84 75c4343d 0×2c45f58
13 0250fd90 77de9812 kernel32!BaseThreadInitThunk+0xe
14 0250fdd0 77de97e5 ntdll!__RtlUserThreadStart+0×70
15 0250fde8 00000000 ntdll!_RtlUserThreadStart+0×1b

The address may be the valid return address from Execution Residue, but may also be completely random, non-executable:

0:008> ub 0×2c45f58
^ Unable to find valid previous instruction for ‘ub 0×2c45f58′

0:008> !address 0×2c45f58

Usage: Free
Base Address: 02bb0000
End Address: 02cb0000
Region Size: 00100000 ( 1.000 MB)
State: 00010000 MEM_FREE
Protect: 00000001 PAGE_NOACCESS

Type: <info not present at the target>

In our case, we have symbol files for ModuleB.dll but they don’t help.

0:008> .sympath+ C:\MemoryDumps\Modules\PDBs

If we have normal Manual Dumps we can compare Stack Trace Collections and take the advantage of existing Thread Posets to get the correct stack trace.

Alternatively, we can either use manual stack trace reconstruction techniques or use Injected Symbols:

0:008> lm
[...]
Unloaded modules:
[...]
68a70000 68ac0000 ModuleB.dll
[…]

0:008> .reload /f /i ModuleB.dll=68a70000
*** WARNING: Unable to verify timestamp for ModuleB.dll

0:008> kL
# ChildEBP RetAddr
00 0250f4b8 76d21775 ntdll!NtWaitForMultipleObjects+0x15
01 0250f554 75c419fc KERNELBASE!WaitForMultipleObjectsEx+0x100
02 0250f59c 75c4268c kernel32!WaitForMultipleObjectsExImplementation+0xe0
03 0250f5b8 75c681fc kernel32!WaitForMultipleObjects+0x18
04 0250f624 75c680bb kernel32!WerpReportFaultInternal+0x186
05 0250f638 75c679b0 kernel32!WerpReportFault+0x70
06 0250f648 75c6792f kernel32!BasepReportFault+0x20
07 0250f6d4 00e21e86 kernel32!UnhandledExceptionFilter+0x1af
08 0250f6f0 75c803cf ModuleA!UnhandledExceptionFilter+0x3d
09 0250f778 77e250d7 kernel32!UnhandledExceptionFilter+0x127
0a 0250f780 77e24fb4 ntdll!__RtlUserThreadStart+0x62
0b 0250f794 77e24e59 ntdll!_EH4_CallFilterFunc+0x12
0c 0250f7bc 77e134a1 ntdll!_except_handler4+0x8e
0d 0250f7e0 77e13473 ntdll!ExecuteHandler2+0x26
0e 0250f804 77e13414 ntdll!ExecuteHandler+0x24
0f 0250f890 77dc0133 ntdll!RtlDispatchException+0x127
10 0250f890 68a8e0ca ntdll!KiUserExceptionDispatcher+0xf
11 0250fd64 68a8f284 ModuleB!foo+0x5a
12 0250fd84 75c4343d ModuleB!bar+0xf4
13 0250fd90 77de9812 kernel32!BaseThreadInitThunk+0xe
14 0250fdd0 77de97e5 ntdll!__RtlUserThreadStart+0×70
15 0250fde8 00000000 ntdll!_RtlUserThreadStart+0×1b

We call this analysis pattern False Frame. Although we have Incorrect Stack Trace, just one stack trace frame is wrong. Sometimes, if there is Coincidental Symbolic Information available we get Coincidental Frames.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 197)

August 23rd, 2020

Sometimes we may want to Flag a message or Activity Region, for example, using Message Annotations. In other cases we may have Activity Regions are sorted by their coordinate-wise inclusion. Or we have inclusion of Message Sets. The analysis pattern name is borrowed from flag filtration in mathematics, where we consider subsets of messages and Activity Regions as subspaces. Dia|gram pictures of Flags may even resemble flags of some countries.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 196)

July 31st, 2020

It is possible to foliate traces into separate traces having the same structure and scale (we also show corresponding Trace Fabric for the original trace):

In the diagram above Trace Foliation was done for message type, for example, error and normal messages. The reverse operation of Trace Mask would produce the same original trace.

Correspondingly Trace Fabric can be foliated too giving rise to “orchestra” representation and vice versa via Trace Mask:

Bars can be added with the help of Silent Messages.

The name of this analysis pattern was also inspired by foliations in mathematics.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 195)

July 31st, 2020

Semantic Field is a set of messages that belong to particular category or subject:

It is different from Trace Field which is a function, not an already prepared codomain of mapping.

Some Semantic Fields may be formed by the analysis of Implementation Discourse, for example using machine learning techniques.

The pattern name was inspired by semantic field in linguistics and came to our attention when reading “German Loanwords in English: An Historical Dictionary” book.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 194)

July 31st, 2020

If we take Combed Trace for Thread of Activity or some Adjoint Thread of Activity and strip other message content like we did for Trace Contour log analysis pattern we get individual braids that form Trace Fabric:

We can also get a stave representation of individual braids after a counter clockwise 90 degree rotation:

Bars can be added with the help of Silent Messages. Conversely, a musical piece can be transformed into some trace.

We mentioned “fabric” metaphor already when we introduced multibraiding.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 193)

July 30th, 2020

If we take Combed Trace for Thread of Activity or some Adjoint Thread of Activity, strip other message content, and then trace all non-empty values we get Trace Contour:


- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 192)

July 20th, 2020

Traces and logs from diverse software systems doing different things may have similar Trace Shape despite completely different message content, especially for specific Threads of Activity or Adjoint Threads of Activity:

This may be apparent when we compare Trace Shape of Quotient Trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 191)

July 19th, 2020

If we have an attribute we can sort messages based on that attribute values and get Sorted Trace. If that attribute is TID or ATID we get the sequence of Threads of Activity or Adjoint Threads of Activity:

If we sort by message types or Message Invariants or some message data we get a sequence of Fibers of Activity.

The diagram above also shows on the right Quotient Trace by message type equivalence after additional sorting inside each Adjoint Thread of Activity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 190)

July 18th, 2020

Causal History messages (black circles) pass through Activity Regions which can be marked as hollow circles:

We call this analysis pattern Trace D’Enfant by analogy with dessin d’enfant in mathematics, a bipartite graph embedded in an oriented surface, so in theory Traces D’Enfants can be studied algebraically.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 189)

July 17th, 2020

The relations between Causal History messages (0-chains) can be abstracted as Causal Chains (1-chains). Two relations can be linked if an endpoint of one is also a beginning point of another:

The relations of 1-chains can be abstracted as 2-chains and so on (n-chains):

We took the idea of relation spaces and chains from already quoted “Discreet Causal Theory” book. Causal chain terminology is also used in philosophy.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 188)

July 16th, 2020

When looking at Causal History we can choose Causal Messages (not necesseraly the top ones):

Causal Messages may not overlap with the trace Defect Group which may not have any causal relevance being only correlation messages.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Analysis Patterns (Part 187)

July 15th, 2020

Trace Paths and Back Traces form Causal History of the log where arrows point in the direction of possible causation:

Here we borrow the notion of causal sets from physics and corresponding mathematics. The left diagramming idea was taken from Discrete Causal Theory book and Hasse diagrams (which is inverted in our picture). Also, such graphs are internal to software narratives compared to the more general external space we proposed earlier.

We omit Time arrow as it is possible to consider general traces and logs with their causality markers.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -