Memory Dump Analysis Services announces the launch of the first Memory Dump Analysis Audit Service:
http://www.dumpanalysis.com/memory-dump-analysis-audit-service
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In any system there is an expected Event Sequence Order as a precondition to its normal behaviour. Any out-of-order events should raise the suspicion bar as they might result or lead to synchronization problems. It need not be a sequence of trace messages from different threads but also between processes, for example, image load events in CDF / ETW traces can indicate a misconfiguration in service startup order. The following diagram depicts a possible pattern scenario:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is the main topic of the forthcoming December Debugged! MZ/PE magazine issue:
Debugged! MZ/PE: Tools for Software Problem Solving, December, 2010 (Paperback, ISBN: 978-1908043023)
The draft front cover symbolizes an element of randomness and chance exceptions in software problem solving.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is a synthesized case study with the simplified ETW trace output based on real events (consider it as an exercise in a software narrative fiction). In a client-server environment a server session was always active regardless of whether a client was active or not. There was a workaround to enable a registry key to force checking for user activity via CheckActivity registry key. Unfortunately this didn’t help and a software trace was recorded for the offline analysis. First, we checked for Basic Facts and found the correspondence that confirmed the registry key setting:
# PID TID Message
[...]
3200 1428 5476 CheckActivity = 1
[...]
There was also Periodic Errors throughout the whole trace fabric:
# PID TID Message
[...]
119097 2468 476 Security check failed
[...]
Although such errors happened in a number of support incidents we see a message text we have never seen and couldn’t even find in our database of incidents (Defamiliarizing Effect):
# PID TID Message
[...]
119090 2468 476 Calling the third party security information provider
119091 2468 476 Get security information: failed
[...]
Therefore we advised to test with the 3rd party security provider disabled. The software story continues…
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca
The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:
- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4
Product information:
Back cover features memory space art image Hot Computation: Memory on Fire.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I’m pleased to announce that MDAA, Volume 4 is available in PDF format:
www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4
It features:
- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art
Its table of contents is available here:
http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf
Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Activity Region pattern highlights “mechanical” and syntactical aspects of trace analysis whereas Focus of Tracing brings attention to changing semantics of trace message flow, for example in Citrix terminal services environment, from logon messages during session initialization to LHC database search. Here is a graphical illustration of this pattern where tracing focus region spans 3 regions of activity:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I have discovered rules that make it possible to devise a memory dump and software trace analysis equivalent of the Periodic Table of Elements in Chemistry. It allows prediction of abnormal software behaviour and structural defects and what patterns to look for after deploying software and collecting its artifacts. More on this is in the next part of these series.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
There are two trace reading practices with techniques borrowed from structuralist and post-structuralist narratology:
1. Close reading
- emphasizes structural patterns
- looks at a software trace as a unity of messages
- searches for similarities, repetitions and contrasts
- reveals code reflections in message texts
2. Deconstructive reading
- reveals subconscious exposed in message texts
- searches for conflicting and absent messages
- looks at a software trace as a disunity of messages from conflicting components
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I detour for MARS expedition. You may also call it Memory Analysis Report System as malware analysis is always exploration of memory (in general). Why is this sudden change of course? After reading Gilles Deleuze I want to broaden the concept of “malware” and give it new orientation and direction of thinking. Beside that I also want new challenges after many years of research in pattern-driven memory dump and software trace analysis of abnormal software behaviour.
You may have also noticed small restructuring (rebranding) of this blog and DumpAnalysis.org headers.
See you there 
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
As a practical example of applying behavioral and structural pattern analysis of computer memory and traces OpenTask plans to publish the following title next year:
The inclusion of victimware is necessary because of the effects of defective malware.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The next pattern with the name borrowed from historiography is Layered Periodization. This periodization of software trace messages includes individual messages, then aggregated messages from threads, then processes as wholes and finally individual computers (in client-server or similar sense). This is best illustrated graphically.
Message layer:
Thread layer (different colors correspond to different TID):
Process layer (different colors correspond to different PID):
Please note that it is also possible to have a periodization based on modules, functions and individual messages but it might be complicated because different threads can enter the same module or function and here other patterns more appropriate like Activity Region, Characteristic Message Block, and Background and Foreground Components.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes, when we find an Anchor Message related to our problem description (for example, a COM port error) we are interested in its evolution throughout a software narrative:
# PID TID Message
[...]
126303 5768 1272 OpenComPort returns Status = 0x0
[...]
231610 3464 1576 OpenComPort returns Status = 0x0
[...]
336535 5768 4292 OpenComPort returns Status = 0×0
[…]
423508 5252 2544 OpenComPort returns Status = 0xc000000f
[…]
531247 5768 5492 OpenComPort returns Status = 0xc000000f
[…]
639039 772 3404 OpenComPort returns Status = 0xc000000f
[…]
Then we can check activity between changes. We call this pattern Message Change.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
I now transition to 8 similar to 7-to-8 transition from user to kernel space (new dialectical quality):
switch(years_at_citrix)
{
case 5:
write_blog_post(”I’ve just passed 5 year mark … “);
wait_for_certificate();
write_blog_post(”Shortly after celebrating 5 years … “);
break;
case 6:
write_blog_post(”Threads in my process run very fast. Not long ago … “);
break;
case 7:
write_blog_post(”Transition to kernel mode and space … “);
break;
case 8:
// … TBD
}
I also abandoned monthly counting started earlier because in kernel space time is counted in clock cycles per quantum unit where the latter is 1/3 of a clock tick (Windows Internals, 5th edition, pp. 406 - 408) :-)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
It is time now to introduce a syntactical notation for memory (dump) and software trace analysis pattern languages (in addition to graphical notation proposed earlier). It should be simple and concise: allow easy grammar with plain syntax and obvious reading semantics. We propose to use capitalized letters for major pattern categories, for example, W for wait chains and D for deadlocks. Then use subscripts (or small letters) for pattern subcategories, for example, Wcs and Dlpc. Several categories and subcategories can be combined by using slash (/), for example, Wcs/Dcs/lpc. Slash notation is better viewed using subscripts:
Wcs/Dcs/lpc
Next part will introduce more categories and propose notational adornments for pattern succession, space differentiation and the inclusion of details in notational sentences.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The choice of webinars below mostly reflects my personal preferences and long time desire to speak on topics like systems thinking, troubleshooting tool design and development. The other topic about BSOD minidump analysis was requested by participants in an ongoing survey. There will be more topics in 2011. All forthcoming webinars will be hosted by Memory Dump Analysis Services. The planning list includes:
Systems Thinking in Memory Dump and Software Trace Analysis
Software Troubleshooting and Debugging Tools: Objects, Components, Patterns and Frameworks with UML
Blue Screen Of Death Analysis Done Right: Minidump Investigation for System Administrators
More detailed information will be available soon.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Another tool that supports adjoint threading in addition to Citrix CDFAnalyzer (see also Debugging Experts magazine article for pictorial description of this concept) is Process Monitor. We can view adjoint threads having common attributes like TID (ordinary threads), PID, operation (function), process name, etc. by using this right click context menu:
For example, this adjoint thread having RegOpenKey as its ATID (Adjoint Thread ID) where we excluded Path, Result and Detail fields for viewing clarity (together these fields can constitute an analogous Message field in TMF traces):
Time of Day Process Name PID TID Operation
[…]
09:33:25.9545410 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9548650 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9550234 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9551656 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9692456 WFICA32.EXE 3588 3496 RegOpenKey
09:33:25.9761325 wfcrun32.exe 852 1148 RegOpenKey
09:33:25.9761912 wfcrun32.exe 852 1148 RegOpenKey
09:33:25.9762295 wfcrun32.exe 852 1148 RegOpenKey
09:33:25.9984547 wfcrun32.exe 852 1148 RegOpenKey
09:33:26.0023831 wfcrun32.exe 852 1148 RegOpenKey
09:33:26.0074675 wfcrun32.exe 852 1148 RegOpenKey
09:33:26.0087191 Explorer.EXE 1292 1032 RegOpenKey
09:33:26.1618595 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1625697 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1632745 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1633924 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1639209 iexplore.exe 1348 2228 RegOpenKey
[…]
So if someone writes a converter from TMF to PML format…
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Inherit a fortune - To get a postmortem artifact like a crash dump.
Examples:
- My program died!
- Did you inherit a fortune?
- Oh, yeah!
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
General Memory Analysis is another name for Memoretics, a discipline that studies memory snapshots including their similarities and differences on different system platforms such as Windows, Linus, Mac OS X, embedded and mobile systems, historical architectures, etc. The analysis of memory helps solve problems in various domains such as software troubleshooting and debugging, computer forensic analysis, etc.
The current focus of interdisciplinary research is to build a unified memory pattern language that covers both behavioral and structural patterns and also to study the possibility of building memory systems from below, not from requirements -> architecture -> design -> implementation -> compilation -> linking -> loading -> execution but from directly modeling and assembling memory systems using memory patterns.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Now it’s time to divide memory analysis patterns discerned so far as mostly abnormal software behavior memory dump and software trace patterns into behavioral and structural catalogues. The goal is to account for normal system-independent structural entities and relationships visible in memory like modules, threads, processes and so on.
The first pattern (and also a super-pattern) we discuss in this part is called Memory Snapshot. It is further subdivided into Structured Memory Snapshot and BLOB Memory Snapshot. Structured sub-pattern includes:
- Contiguous memory dump files with artificially generated headers (for example, physical or process virtual space memory dump)
- Software trace messages with imposed internal structure
BLOB sub-pattern variety includes address range snapshots without any externally imposed structure, for example, saved by .writemem WinDbg command or ReadProcessMemory API and contiguous buffer and raw memory dumps saved by various memory acquisition tools.
Behavioral patterns that relate to Memory Snapshot pattern are:
I strive initially to publish at least one such pattern every day to fill the gap of normal patterns in memory analysis and later add more multi-platform details and examples from other platforms like Linux, Mac OS X, embedded and selected important historical architectures.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -