With the new year starts the new initiative to integrate traditional multidisciplinary debugging approaches and methodologies with multiplatform pattern-driven software problem solving, unified debugging patterns, best practices in memory dump analysis and software tracing, computer security, economics, and the new emerging trends I’m going to write about during this year.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In these post series we are going to discuss the best practices for software tracing implementation including appropriate patterns and their links to software trace analysis patterns. The first one is called Period Timestamp where the start and the end time (and the date if necessary) are recorded in the trace file. This helps in Inter-Correlation and News Value analysis between several different trace types. For example, in one scenario, we had WindowHistory and MessageHistory logs. We identified a problem in the former log as happening at this time:
Handle: 00010196 Class: "ClassA" Title: "TitleA"
Captured at: 13:36:30:533
[…]
However, when we looked at the latter trace to search for specific window messages posted or sent before that time we saw that the recording started later than the former event:
Start time: 13:36:35:830
Period timestamps are necessary to distinguish Incomplete History from Truncated Trace where in the former case the absence of expected trace message is due to some problem.
From a unified debugging patterns perspective we have this sequence fragment:
Implementation Patterns: Period Timestamp
Usage Patterns: Trace Simultaneously
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
News Value is a pattern that assigns relative importance to software traces for problem solving purposes especially when related to problem description, recent incidents and timestamps of other supporting artifacts (memory dumps, other traces, etc.). For example, in one scenario, an ETW trace was provided with 3 additional log files:
# Source PID TID Date Time Message
0 Header 1260 1728 12/14/2010 06:48:56.289 ?????
[…]
215301 Unknown 640 808 12/14/2010 07:22:57.508 ????? Unknown( 16): GUID=[…] (No Format Information found).
// LogA
05/11/10 18:28:15.1562 : Service() - entry
[...]
14/12/10 10:31:58.0381 : Notification: sleep
* Start of new log *
14/12/10 10:34:38.4687 : Service() - entry
[…]
14/12/10 11:53:35.2729 : Service.CleanUp complete
* Start of new log *
14/12/10 11:56:11.7031 : Service() - entry
[…]
14/12/10 15:25:23.3004 : Notification: sleep
// LogB
[ 1] 12/14 10:34:29:890 Entry: ctor
[…]
[ 2] 12/14 11:53:30:866 Exit: COMServer.Server.DeleteObject
// LogC
[ 1] 12/14 11:56:03:359 Entry: ctor
[…]
[ 20] 12/14 15:30:20:110 Exit: Kernel32.Buffer.Release
From the description of the problem we expected LogB and LogC to be logs from two subsequent process executions where the first launch fails (LogB) and the second launch succeeds (LogC). Looking at their start and end times we see that they make sense from the problem description perspective but we have to dismiss ETW trace and most of LogA as recorded earlier and having no value for Inter-Correlation analysis of the more recent logs. We also see that portions of LogA overlap with LogB and LogC and therefore having analysis value for us.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
As the new decade is approaching (2011-2020) we would like to make a few previews and predictions:
- Increased complexity of software will bring more methods from biological, social sciences and humanities in addition to existing methods of automated debugging and computer science techniques
- Focus on first fault software problem solving (when aspect)
- Focus on pattern-driven software problem solving (how aspect)
- Fusion of debugging and malware analysis into a unified structural and behavioral pattern framework
- Visual debugging, memory and software trace visualization techniques
- Software maintenance certification
- Focus on domain-driven troubleshooting and debugging tools as a service (debugware TaaS)
- Focus on security issues related to memory dumps and software traces
- New scripting languages and programming language extensions for debugging
- The maturation of the science of memory snapshots and software traces (memoretics)
Imagining is not not limited to the above and more to come and explain in the forthcoming parts.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
If we look at any non-trivial trace we would see different Implementation Discourses. Components are written in different languages and adhere to different runtime environments, binary models and interface frameworks. All these implementation variations influence the structure, syntax and semantics of trace messages. For example, .NET debugging traces differ from file system driver or COM debugging messages. Therefore we establish the new field of Software Trace Linguistics as a science of software trace languages. Some parallels can be drawn here towards software linguistics (the science of software languages) although we came to that conclusion independently while thinking about applying “ethnography of speaking” to software trace narration. More on this in the following posts.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We now start unifying software behavior analysis patterns with debugging architecture, design, implementation and usage. This is analogous to software construction where a problem analysis leads to various software engineering phases. The important difference here is the addition of debugging usage patterns. Let’s look at an example (we discuss suggested patterns later):
- Analysis Patterns
- Architectural Patterns
Debug Event Subscription / Notification
- Design Patterns
Punctuated Execution
- Implementation Patterns
Breakpoint (software and hardware)
- Usage Patterns
Kernel vs. user space breakpoints
To differentiate this systematic approach from the various published ad hoc debugging patterns we call it Unified Debugging Pattern Language. ADI parts can also correspond to various DebugWare patterns where we provide a mapping later.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
MAaaS includes 2 complementary DA+TA services:
1. Dump Analysis as a Service (DAaaS)
2. Trace Analysis as a Service (TAaaS)
Memory Dump Analysis Services is the first organization to provide such a service at an audit and certification levels.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Memory Dump Analysis Services announces the launch of the first Memory Dump Analysis Audit Service:
http://www.dumpanalysis.com/memory-dump-analysis-audit-service
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In any system there is an expected Event Sequence Order as a precondition to its normal behaviour. Any out-of-order events should raise the suspicion bar as they might result or lead to synchronization problems. It need not be a sequence of trace messages from different threads but also between processes, for example, image load events in CDF / ETW traces can indicate a misconfiguration in service startup order. The following diagram depicts a possible pattern scenario:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is the main topic of the forthcoming December Debugged! MZ/PE magazine issue:
Debugged! MZ/PE: Tools for Software Problem Solving, December, 2010 (Paperback, ISBN: 978-1908043023)
The draft front cover symbolizes an element of randomness and chance exceptions in software problem solving.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is a synthesized case study with the simplified ETW trace output based on real events (consider it as an exercise in a software narrative fiction). In a client-server environment a server session was always active regardless of whether a client was active or not. There was a workaround to enable a registry key to force checking for user activity via CheckActivity registry key. Unfortunately this didn’t help and a software trace was recorded for the offline analysis. First, we checked for Basic Facts and found the correspondence that confirmed the registry key setting:
# PID TID Message
[...]
3200 1428 5476 CheckActivity = 1
[...]
There was also Periodic Errors throughout the whole trace fabric:
# PID TID Message
[...]
119097 2468 476 Security check failed
[...]
Although such errors happened in a number of support incidents we see a message text we have never seen and couldn’t even find in our database of incidents (Defamiliarizing Effect):
# PID TID Message
[...]
119090 2468 476 Calling the third party security information provider
119091 2468 476 Get security information: failed
[...]
Therefore we advised to test with the 3rd party security provider disabled. The software story continues…
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca
The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:
- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4
Product information:
Back cover features memory space art image Hot Computation: Memory on Fire.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I’m pleased to announce that MDAA, Volume 4 is available in PDF format:
www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4
It features:
- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art
Its table of contents is available here:
http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf
Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Activity Region pattern highlights “mechanical” and syntactical aspects of trace analysis whereas Focus of Tracing brings attention to changing semantics of trace message flow, for example in Citrix terminal services environment, from logon messages during session initialization to LHC database search. Here is a graphical illustration of this pattern where tracing focus region spans 3 regions of activity:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I have discovered rules that make it possible to devise a memory dump and software trace analysis equivalent of the Periodic Table of Elements in Chemistry. It allows prediction of abnormal software behaviour and structural defects and what patterns to look for after deploying software and collecting its artifacts. More on this is in the next part of these series.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
There are two trace reading practices with techniques borrowed from structuralist and post-structuralist narratology:
1. Close reading
- emphasizes structural patterns
- looks at a software trace as a unity of messages
- searches for similarities, repetitions and contrasts
- reveals code reflections in message texts
2. Deconstructive reading
- reveals subconscious exposed in message texts
- searches for conflicting and absent messages
- looks at a software trace as a disunity of messages from conflicting components
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I detour for MARS expedition. You may also call it Memory Analysis Report System as malware analysis is always exploration of memory (in general). Why is this sudden change of course? After reading Gilles Deleuze I want to broaden the concept of “malware” and give it new orientation and direction of thinking. Beside that I also want new challenges after many years of research in pattern-driven memory dump and software trace analysis of abnormal software behaviour.
You may have also noticed small restructuring (rebranding) of this blog and DumpAnalysis.org headers.
See you there 
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
As a practical example of applying behavioral and structural pattern analysis of computer memory and traces OpenTask plans to publish the following title next year:
The inclusion of victimware is necessary because of the effects of defective malware.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The next pattern with the name borrowed from historiography is Layered Periodization. This periodization of software trace messages includes individual messages, then aggregated messages from threads, then processes as wholes and finally individual computers (in client-server or similar sense). This is best illustrated graphically.
Message layer:
Thread layer (different colors correspond to different TID):
Process layer (different colors correspond to different PID):
Please note that it is also possible to have a periodization based on modules, functions and individual messages but it might be complicated because different threads can enter the same module or function and here other patterns more appropriate like Activity Region, Characteristic Message Block, and Background and Foreground Components.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes, when we find an Anchor Message related to our problem description (for example, a COM port error) we are interested in its evolution throughout a software narrative:
# PID TID Message
[...]
126303 5768 1272 OpenComPort returns Status = 0x0
[...]
231610 3464 1576 OpenComPort returns Status = 0x0
[...]
336535 5768 4292 OpenComPort returns Status = 0×0
[…]
423508 5252 2544 OpenComPort returns Status = 0xc000000f
[…]
531247 5768 5492 OpenComPort returns Status = 0xc000000f
[…]
639039 772 3404 OpenComPort returns Status = 0xc000000f
[…]
Then we can check activity between changes. We call this pattern Message Change.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -