Diagnostics Science

All areas of human activity involve the use of diagnostics. Proper diagnostics identifies the right problems to solve. We are now a part of a non-profit organization dedicated to the developing and promoting the application of such diagnostics: systemic and pattern-oriented (pattern-driven and pattern-based).

Encyclopedia of Crash Dump Analysis Patterns

Available in PDF format from Software Diagnostics Services

This reference reprints with corrections, additional comments, and classification 326 alphabetically arranged and cross-referenced memory analysis patterns originally published in Memory Dump Analysis Anthology volumes 1 – 8. This pattern catalog is a part of pattern-oriented software diagnostics, forensics, and prognostics developed by Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org). Most of the patterns are illustrated with examples for WinDbg from Debugging Tools for Windows with a few examples from Mac OS X for GDB.

Product information:

  • Title: Encyclopedia of Crash Dump Analysis Patterns: Detecting Abnormal Software Structure and Behavior in Computer Memory
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 24.6 x 18.9
  • Paperback: 1,058 pages
  • Publisher: OpenTask (March 2015)
  • ISBN-13: 978-1-906717-21-6

Table of Contents

Memory Dump and Live Memory Visualization and Picture Extraction

This is a reprint of several articles with permission from Memory Dump Analysis Anthology, Volume 1.

As the first step towards Memory Dump Tomography (Memory Dump Analysis Anthology, Volume 1, page 522) we created a small program that interprets a memory dump as a picture. We can visualize crash dumps with it. The tool is available for free download.

Simply run it from the command prompt and specify full paths to a dump file and an output BMP file. The memory dump file will be converted by default into true color, 32 bits-per-pixel bitmap. We can specify other values: 8, 16 and 24.

C:\Dump2Picture>Dump2Picture.exe
Dump2Picture version 1.0
Written by Dmitry Vostokov, 2007
Usage: Dump2Picture dumpfile bmpfile [8|16|24|32]

For example:

C:\Dump2Picture>Dump2Picture.exe MEMORY.DMP MEMORY.BMP 8
Dump2Picture version 1.0
Written by Dmitry Vostokov, 2007
MEMORY.BMP
MEMORY.DMP
1 file(s) copied.

Below are some screenshots of bitmap files created by the tool. We can think about them as visualized kernel or user address spaces.

Vista kernel memory dump (8 bits-per-pixel):

Vista kernel memory dump (16 bits-per-pixel):

Vista kernel memory dump (24 bits-per-pixel):

Vista kernel memory dump (32 bits-per-pixel):

Notepad process user memory dump (8 bits-per-pixel):

Notepad process user memory dump (16 bits-per-pixel):

Notepad process user memory dump (24 bits-per-pixel):

Notepad process user memory dump (32 bits-per-pixel):

Mspaint process user memory dump (32 bits-per-pixel):

Mspaint process user memory dump after loading "Toco Toucan.jpg" from Vista Sample Pictures folder (32 bits-per-pixel):

Dump2Picture can be used to explore memory leaks visually. We created the following small program in Visual C++ that leaks 64Kb every second:

#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    printf("%x\n", (UINT_PTR)malloc(0xFFFF));
    Sleep(1000);
  }
  return 0;
}

Then we sampled 3 dumps at 7Mb, 17Mb and 32Mb process virtual memory size and converted them as 16 bits-per-pixel bitmaps. On the pictures below we can see that the middle black memory area grows significantly. Obviously malloc function allocates zeroed memory and therefore we see black color.

7Mb process memory dump:

17Mb process memory dump:

32Mb process memory dump:

If we zoom in the black area we would see the following pattern:

Colored lines inside are heap control structures that are created for every allocated block of memory. If this is correct then allocating smaller memory blocks would create a hatched pattern. And this is true indeed. The following program leaks 256 byte memory blocks:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    printf("%x\n", (UINT_PTR)malloc(0xFF));
    Sleep(1000/0xFF);
  }
  return 0;
}

The corresponding process memory picture and zoomed heap area are the following:

Making allocations 4 times smaller makes heap area to look hatched, and zoomed picture is more densely packed by heap control structures:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
  while (true)
  {
    printf("%x\n", (UINT_PTR)malloc(0xFF/4));
    Sleep((1000/0xFF)/4);
  }
  return 0;
}

Here is another example. One service was increasing its memory constantly. The crash dump picture shows huge hatched dark region in the middle:

and if we zoom in this region:

Because the pattern and allocation size look uniform it could be the true heap memory leak for some operation that allocates constant size buffers. After opening the dump and looking at heap segments that had grown the most we see the same allocation size indeed:

0:000> !.\w2kfre\ntsdexts.heap -h 5
HEAPEXT: Unable to get address of NTDLL!NtGlobalFlag.
Index Address Name Debugging options enabled
1: 00140000
2: 00240000
3: 00310000
4: 00330000
5: 00370000
Segment at 00370000 to 00380000 (00010000 bytes committed)
Segment at 01680000 to 01780000 (00100000 bytes committed)
Segment at 019C0000 to 01BC0000 (00200000 bytes committed)
Segment at 01BC0000 to 01FC0000 (00400000 bytes committed)
Segment at 01FC0000 to 027C0000 (00800000 bytes committed)
Segment at 027C0000 to 037C0000 (01000000 bytes committed)
Segment at 037C0000 to 057C0000 (02000000 bytes committed)

Segment at 057C0000 to 097C0000 (00155000 bytes committed)
...
...
...
057B96E0: 01048 . 01048 [07] - busy (1030), tail fill
057BA728: 01048 . 01048 [07] - busy (1030), tail fill
057BB770: 01048 . 01048 [07] - busy (1030), tail fill
057BC7B8: 01048 . 01048 [07] - busy (1030), tail fill
057BD800: 01048 . 01048 [07] - busy (1030), tail fill
057BE848: 01048 . 01048 [07] - busy (1030), tail fill
057BF890: 01048 . 00770 [14] free fill
Heap entries for Segment07 in Heap 370000
057C0040: 00040 . 01048 [07] - busy (1030), tail fill
057C1088: 01048 . 01048 [07] - busy (1030), tail fill
057C20D0: 01048 . 01048 [07] - busy (1030), tail fill
057C3118: 01048 . 01048 [07] - busy (1030), tail fill
057C4160: 01048 . 01048 [07] - busy (1030), tail fill
057C51A8: 01048 . 01048 [07] - busy (1030), tail fill
...
...
...

An alternative to converting memory dumps to picture files is to save a memory range to a binary file and then convert it to a BMP file. Thus we can view the particular DLL or driver mapped into address space, heap or pool region, including live memory:

To save a memory range to a file use WinDbg .writemem command:

.writemem d2p-range.bin 00800000 0085e000

or

.writemem d2p-range.bin 00400000 L20000

We wrote a WinDbg script that saves a specified memory range and then calls a shell script that automatically converts the saved binary file to a BMP file and then runs a picture viewer registered for .bmp extension.

The WinDbg script code (mempicture.txt):

.writemem d2p-range.bin ${$arg1} ${$arg2}
.if (${/d:$arg3})
{
  .shell -i- mempicture.cmd d2p-range ${$arg3}
}
.else
{
  .shell -i- mempicture.cmd d2p-range
}

The shell script (mempicture.cmd):

dump2picture %1.bin %1.bmp %2
%1.bmp

Because WinDbg installation folder is assumed to be the default directory for both scripts and Dump2Picture.exe they should be copied to the same folder where windbg.exe is located. On our system it is

C:\Program Files\Debugging Tools for Windows

Both scripts are now included in Dump2Picture package available for free download.

To call the script from WinDbg use the following command:

$$>a< mempicture.txt Range [bits-per-pixel]

where Range can be in Address1 Address2 or Address Lxxx format, bits-per-pixel can be 8, 16, 24 or 32. By default it is 32.

For example, we loaded a complete Windows x64 memory dump and visualized HAL (hardware abstraction layer) module:

kd> lm
start end module name
fffff800`00800000 fffff800`0085e000 hal
fffff800`01000000 fffff800`0147b000 nt
fffff97f`ff000000 fffff97f`ff45d000 win32k
...
...
...

kd> $$>a< mempicture.txt fffff800`00800000 fffff800`0085e000
Writing 5e001 bytes...

C:\Program Files\Debugging Tools for Windows>dump2picture d2p-range.bin d2p-range.bmp

Dump2Picture version 1.1
Written by Dmitry Vostokov, 2007

d2p-range.bmp
d2p-range.bin
1 file(s) copied.

C:\Program Files\Debugging Tools for Windows>d2p-range.bmp
<.shell waiting 10 second(s) for process>
.shell: Process exited
kd>

and Windows Picture and Fax Viewer application was launched and displayed the following picture:

Software Trace and Log Analysis: A Pattern Reference

The following direct links can be used to order the book now:

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Also available in PDF format from Software Diagnostics Services

General trace and log analysis patterns allow application of uniform problem detection and solving approach across diverse software environments. This pattern language covers any execution artifact from a small debugging trace to a distributed log with billions of messages from hundreds of computers, thousands of software components, threads, and processes. Pattern-oriented trace and log analysis is applicable to troubleshooting and debugging Windows, Mac OS X, Linux, FreeBSD, Android, iOS, z/OS, and any other possible computer platform. Its pattern catalog is a part of pattern-oriented software diagnostics, forensics, and prognostics developed by Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org). This reference reprints with corrections 100 patterns originally published in Memory Dump Analysis Anthology volumes 3 - 8a and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog). Full-color diagrams accompany most pattern descriptions.

Product information:

  • Title: Software Trace and Log Analysis: A Pattern Reference
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 21.6 x 14.0
  • Paperback: 192 pages
  • Publisher: OpenTask (February 2015)
  • ISBN-13: 978-1-908043-80-1

Table of Contents

Special and General Trace and Log Analysis

Most software traces include message timestamps or have an implicit time arrow via sequential ordering. We call such traces Special. The analysis is special too because causality is easily seen. Typical examples of analysis patterns here are Discontinuity, Time Delta, Event Sequence Order, Data Flow (see time dependency markers ↓ in the training course reference), and more recently added patterns such as Back Trace, Timeout, Milestones, and Event Sequence Phase. Inter- and Intra-Correlation analysis is also easy.

On the other side, there are plenty of software logs or digital media artefacts with “chaotic” records where time arrow is missing or only partial. Typical examples are debugger logs from WinDbg debugger from Microsoft Debugging Tools for Windows or logs from debugging sessions on other platforms. Such logs may contain global ordering such as the list of processes and threads (Last Object memory analysis pattern) interspaced with local pockets of stack traces that have reversed ordering. Some logging output may not have any ordering or timing information whatever.

In a more general case, logging may be completely arbitrarily. Typical example is raw stack analysis and its Rough Stack Trace and Past Stack Trace patterns.

We call such traces General. The main task of general trace analysis is to recover causality. It may be possible if another analysis pattern is introduced called Causality Markers. The prototypes of such a pattern are various wait chains, Waiting Thread Time memory analysis pattern and its process memory dump equivalent.

Diagnostics, Forensics, Prognostics: The Copernican Revolution

Our pattern language system for software diagnostics was originally tool-centric, devised from recurrent analysis problems on Windows platforms when using WinDbg from Microsoft Debugging Tools for Windows. We now call it “Ptolemaic”.

Later we realized that the same system is possible to apply to other platforms and tools, and we proved it with the publication of the book for GDB. When the debugging tool changed on Mac OS X we ported the same book to different LLDB debugger. With the presenting of a pattern language for memory forensics, we extended the same system to other memory analysis tools in principle. Our pattern language system became pattern-centric. We call it “Copernican”. In addition, trace and log analysis pattern system was devised as pattern-centric from the ground up based on software Narratology and applied to all software execution artifacts including network traces.

Although this paradigm shift was evolutionary for us, we call it revolutionary for the whole discipline of software diagnostics, forensics, and prognostics because of previously dominant tool-centric systems.

Pattern-Oriented Network Forensics

In February, 2015 we plan a lecture which introduces a pattern language for network forensics - investigation of captured network traces for patterns of abnormal network traffic behavior. It provides a unified language for discussing and communicating detection and analysis results, a base language for checklists, and an aid in accelerated learning. Pattern-oriented network forensics is based on a synthesis of pattern-oriented network trace analysis with malware narrative approach including malnarratives and complements pattern-oriented memory forensics.

Pattern-Oriented Debugging Process

Modern debugging is complex and usually distributed across organizations involving many persons and teams:

We propose fully pattern-oriented debugging process that takes into account the integral role of software diagnostics and software construction patterns.

Whenever we have a software incident, we usually start with a small number of Elementary Software Diagnostics Patterns to identify what kind of software execution artifacts to collect. Such artifacts may include memory dumps and logs for postmortem debugging but can also be running software itself, the so-called live debugging scenario. These patterns can be also called Software Diagnostics Analysis Patterns reflecting the fact that we need to analyze what we further need to do before doing software diagnostics itself.

Based on artifacts we identify patterns of software behavior such memory and trace analysis patterns. We call these patterns Software Diagnostics Usage Patterns.

Such usage patterns can be also called Debugging Analysis Patterns because we need to diagnose the right problem before doing any debugging.

Specific techniques reused across different software diagnostics and debugging scenarios we name Software Diagnostics and Debugging Implementation Patterns. There can also be Debugging Usage Patterns as reusable debugging scenarios.

Previously we introduced Unified Debugging Patterns (Analysis, Architecture, Design, Implementation, and Usage) to which we would like to add Debugging Presentation Patterns and similar pattern stack for software diagnostics:

These pattern categories are usually already in existence and are important only when we develop new software diagnostics and debugging infrastructures and tools. For example, Patterns-View-Controller architectural pattern.

Examples of Elementary Software Diagnostics (Software Diagnostics Analysis) Patterns, Software Diagnostics Usage (Debugging Analysis) Patterns, and Debugging Implementation Patterns can be found in Accelerated Windows Debugging3 book.

Memory Dump Analysis Anthology, Volume 8a

Now the paperback version is in full color!

New! Available for Safari Books Online subscribers

The following direct links can be used to order the book now:

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Also available in PDF format from Software Diagnostics Services

Contains revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, debugging, crash dump analysis, memory forensics, software trace and log analysis written in June 2014 - November 2014 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, reverse engineers, malware and memory forensics analysts. This volume is fully cross-referenced with volumes 1 - 7 and features:

- 19 new crash dump analysis patterns
- 10 new software log and trace analysis patterns
- Introduction to malnarratives and higher-order pattern narratives
- Introduction to pattern language for performance analysis
- Introduction to pattern-oriented debugging process

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 8a
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 150 pages
  • Publisher: OpenTask (December 2014)
  • ISBN-13: 978-1-908043-53-5

Table of Contents
Errata

Software Diagnostics Professional Certification

Software Diagnostics Services offers a certification in pattern-oriented software diagnostics for software technical support and escalation engineers, software engineers and quality assurance engineers with the following assessment areas:

  • pattern-oriented memory dump analysis
  • pattern-oriented software trace and log analysis

The focus is on unintentional software behaviour such related to crashes, resource consumption (CPU, memory leaks), and hangs. For intentional software behaviour there is Memory Forensics Professional certification under development. The Windows track tests the ability to recognise patterns using the following analysis tools: WinDbg from Microsoft Debugging Tools for Windows and Process Monitor.

Please visit Software Diagnostics Services for more details.

Memory Forensics Professional Certification

Software Diagnostics Services offers a certification in pattern-oriented memory forensics for digital forensics and incident response professionals, reverse engineers and security researchers with the following assessment areas:

  • pattern-oriented malware detection and analysis
  • pattern-oriented reverse engineering

The focus is on intentional software behaviour such related to malware and rootkits. For unintentional software behaviour there is Software Diagnostics Professional certification. The Windows track tests the ability to recognise ADDR and malware analysis patterns using the following analysis tool: WinDbg from Microsoft Debugging Tools (future versions of this certification may add other tools). It has the same features and process as Software Diagnostics Professional certification.

Please visit Software Diagnostics Services for more details.

Book: Principles of Memory Dump Analysis

New! Available for Safari Books Online subscribers

The following direct links can be used to order the book now:

Buy Hardcover from Amazon

Buy Hardcover from Barnes & Noble

Also available for sale in PDF format from Software Diagnostics Services.

This is a collection of Software Diagnostics Services webinar transcripts about memory dump analysis methodology developed by Software Diagnostics Institute. Includes 6 seminars on physical memory dump analysis, cloud memory dump analysis, patterns, tools, processes and best practices for software trace and memory dump analysis, pattern-oriented software forensics, a pattern language for memory forensics, and mobile software diagnostics.

  • Title: Principles of Memory Dump Analysis: The Collected Seminars
  • Authors: Dmitry Vostokov, Software Diagnostics Services
  • Publisher: OpenTask (September 2014)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Hardback: 284 pages
  • ISBN-13: 978-1906717667

Software Trace and Memory Dump Analysis

The following direct links can be used to order the book now:

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Available for SkillSoft Books24x7 subscribers

This is a full-colour transcript of Software Diagnostics Services (former Memory Dump Analysis Services) lecture delivered at E2EVC Virtualization Conference in 2011 which introduces an analysis methodology for software execution artefacts.

  • Title: Software Trace and Memory Dump Analysis: Patterns, Tools, Processes and Best Practices
  • Author: Dmitry Vostokov, Software Diagnostics Services
  • Publisher: OpenTask (September 2014)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 28 pages
  • ISBN-13: 978-1908043238

Malnarratives

A malnarrative is an intentionally modified narrative for malicious purposes. This word comes from the so-called malware narratives and their patterns. Malware narratives are just software traces and logs (for example, system logs and network traces) that contain diagnostic indicators (signs) pointing to possible or actual malware presence and execution. Therefore, malware narrative analysis patterns are based on general software trace and log analysis patterns as a part of pattern-oriented software diagnostics and forensics. Whereas, malware narratives result from planned alteration of structure and behaviour of software to serve malicious purposes with resulting narratives incidentally revealing malware, malnarratives are planned alterations of narratives themselves. Because software narratives are based on software narratology (which is an application of general narratology) the extensive trace and log analysis pattern catalogue (more than 90 patterns at the time of this writing) can be used to analyse and detect such patterns in non-software narratives. For example, it can be used for analysis of cyberspace narratives such as social media narratives (Facebook, Twitter, and LinkedIn) and even traditional media narratives such as news, stories, and books. Such pattern-oriented analysis of malnarratives can be used not only in security but also in intelligence analysis (MDAA, Volume 6) and information operations (IO), for example, in information warfare (IW).

A Pattern Language for Performance Analysis

We introduce a new software diagnostics pattern sub-catalogue based on trace and log analysis patterns. It now includes the following performance analysis patterns (with more patterns added soon):

Counter Value
Global Monotonicity
Constant Value
...

Many general log analysis patterns based on software narratology are applicable to performance monitoring logs because they structure analysis data and corresponding thought process:

Adjoint Thread (can be visualized via different colours on a graph)
Focus of Tracing
Characteristic Message Block (for graphs)
Activity Region
Significant Event
and many others.

The goal is to discern, describe, and classify general regularities and their interactions in captured performance data including analysis approaches reused across different operating systems, products, and their performance monitoring and analysis tools. Such Pattern-Oriented Performance Analysis as a part of Pattern-Oriented Software Diagnostics includes pattern-driven, pattern-based, and systemic parts.

Forthcoming Windows Debugging Notebook: Essential Concepts and Tools

This is a reference book for technical support and escalation engineers troubleshooting and debugging complex software issues. The book is also invaluable for software maintenance and development engineers debugging Windows applications and services.

  • Title: Windows Debugging Notebook: Essential Concepts and Tools
  • Authors: Malcolm McCaffery, Dmitry Vostokov
  • Language: English
  • Product Dimensions: 19.8 x 12.9 cm
  • ISBN-13: 978-1-908043-16-0
  • Publisher: OpenTask (November 2014)
  • Paperback: 256 pages

Pattern-Oriented Memory Forensics

The following direct links can be used to order the book now:

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

This is a full-colour transcript of a lecture which introduces a pattern language for memory forensics - investigation of past software behaviour in memory snapshots. It provides a unified language for discussing and communicating detection and analysis results despite the proliferation of operating systems and tools, a base language for checklists, and an aid in accelerated learning. The lecture has a short theoretical part and then illustrates various patterns seen in crash dumps by using WinDbg debugger from Microsoft Debugging Tools for Windows.

Also available in PDF and/or print format as a part of Memory Forensics Training Pack from Software Diagnostics Services.

  • Title: Pattern-Oriented Memory Forensics: A Pattern Language Approach
  • Author: Dmitry Vostokov, Software Diagnostics Institute, Software Diagnostics Services
  • Publisher: OpenTask (September 2014)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 98 pages
  • ISBN-13: 978-1908043764

Presentation Slides

The Timeless Way of Diagnostics

Paraphrasing 2 classical books of architecture written by Christopher Alexander, et al. "The Timeless Way of Building" and "A Pattern Language: Towns, Buildings, Construction" we would like to introduce the complete restructuring of multivolume Memory Dump Analysis Anthology into the projected 10 volume "A Pattern Language for Software Diagnostics, Forensics, and Prognostics: Memory, Traces, Deconstruction". The first volume is planned for the beginning of October (ISBN: 978-1908043818) and then we plan to release additional volume every month until next Summer. The reference will have better browsing and cross-referencing format, additional examples and case studies. It will incorporate comments and new pattern knowledge acquired since the first patterns were described 8 years ago. The new edition will cover only patterns and will not include additional content found in Memory Dump Analysis Anthology such as philosophy and art. Here's the preliminary front cover based on Software Diagnostics Institute logo:

Memory Dump Analysis Anthology will continue to be released with Volume 8 planned for 2015 and include up to date research from Software Diagnostics Institute and additional topics not included in "A Pattern Language for Software Diagnostics, ...".

Higher-Order Pattern Narratives (Analysing Diagnostic Analysis)

A pattern narrative in software narratology means a narrative where messages or log entries are patterns from pattern catalogues. These can be either domain specific patterns, or general trace and log analysis patterns such as Discontinuity, Activity Region, Significant Event, Macrofunction, and Back Trace. Generally, a software pattern narrative is a narrative constructed from software execution artefacts such as logs and memory dumps during their analysis. This is different from the usual meaning of a pattern narrative in narratology and literary criticism where we have a narrative which is a pattern itself like Master Trace analysis pattern. So in our case it is a narrative of patterns and not a narrative which is a pattern. The following picture illustrates the correspondence between a software trace example and its software pattern narrative:

A pattern narrative can be further analysed for any missing patterns using pattern sequences and schemes.

By a second-order narrative we mean a narrative about narrative such as the analysis of original narrative (a first-order narrative). For example, a transformation of a software log into its pattern narrative equivalent (the so called analysis narrative) is a second order narrative. It has its own time sequence (called Analysis Time, TA) where certain patterns are diagnosed out-of-order of their appearance in the resulting pattern narrative. In such a narrative additional patterns may be included that were diagnosed initially but were later replaced or eliminated. The latter property shows that second-order narratives are not simply rearranged plots of the same story (fabula). In the case of generalized memory narratives and hybrid artefacts such as memory dumps we also have analysis narratives. The following picture shows the analysis narrative used to construct software pattern narrative from the previous picture example. We see what patterns were found first:

Such second-order narratives can be analysed further and give rise to third-order narratives and in general to higher-order narratives. The following diagram shows the relationship between a software narrative (order N), its pattern narrative (order N) and analysis narrative (order N+1):

The same principles of software narratology and its analysis patterns can be applied here. Discontinuities, Time Deltas and other patterns can be analysed to find out analysis difficulties which might require further training (such as in domain specific knowledge) and analysis tool development for subsequent computer assistance. For example, if we compare TAs in the analysis report above we find out that there was significant Time Delta pattern before Back Trace pattern was found leading to the first Error Message pattern from a different Thread of Activity. It took some time for an analyst to get an idea to look for specific Data Flow pattern and construct Back Trace (probably by looking at source code).

If you are new to Software Narratology please find this introduction: http://www.patterndiagnostics.com/Introduction-Software-Narratology-mate...

Native Memory Forensics

Among different approaches to memory forensics (investigation of past system or process structure and behaviour recorded in memory snapshots) native memory forensic analysis is done using native OS debuggers such as WinDbg from Debugging Tools for Windows or GDB (Linux) or GDB/LLDB (Mac OS X). Such approach is an integral part of software diagnostics (investigation of signs of software structure and behaviour in software execution artefacts) and was introduced as a part of pattern-oriented software forensics:

http://www.patterndiagnostics.com/pattern-oriented-software-forensics-ma...

Software Diagnostics Services offers a comprehensive self-paced training course in native memory forensics for Windows platforms (Windows XP, Windows Vista, Windows 7, Windows 8, Windows RT, Windows Server) using WinDbg and memory dumps in hands-on exercises:

http://www.patterndiagnostics.com/memory-forensics-pack

Their training course (which also includes malware and rootkit detection, disassembly and reversing as an integral part of forensic investigation) teaches various pattern languages (such as memory analysis pattern language, malware analysis patterns, and ADDR patterns) that can be used with other memory forensic analysis tools.

Memory Dump Analysis Anthology: Color Supplement for Volumes 6-7

The following direct links can be used to order the book now:

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

This is a supplemental volume of 150 full color illustrations from Memory Dump Analysis Anthology: revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, debugging, crash dump analysis, software narratology, software trace and log analysis, malware analysis and memory forensics written in November 2011 - May 2014 for software engineers developing and maintaining software products, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, malware analysts, reverse engineers, digital forensics analysts, computer security and cyber warfare intelligence professionals, computer scientists, conceptual digital artists, and philosophers. Unique in its breadth, depth, and scope it offers unprecedented insight into the world of software behavior and draws profound engineering, scientific, artistic, and philosophical implications.

  • Title: Memory Dump Analysis Anthology: Color Supplement for Volumes 6-7
  • Author: Dmitry Vostokov
  • Publisher: OpenTask (June 2014)
  • Language: English
  • Product Dimensions: 21.6 x 14.0
  • Paperback: 200 pages
  • ISBN-13: 978-1908043740

Table of Contents

Memory Dump Analysis Anthology, Volume 7

New! Available for Safari Books Online subscribers

The following direct links can be used to order the book now:

Buy Paperback or Hardcover from Amazon

Buy Paperback or Hardcover from Barnes & Noble

Also available in PDF format from Software Diagnostics Services

Contains revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, debugging, crash dump analysis, software trace and log analysis, malware analysis and memory forensics written in November 2011 - May 2014 for software engineers developing and maintaining products on Windows (WinDbg) and Mac OS X (GDB) platforms, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, malware analysts, reverse engineers, and memory forensics analysts. The seventh volume features:

- 66 new crash dump analysis patterns
- 46 new software log and trace analysis patterns
- 18 core memory dump analysis patterns for Mac OS X and GDB
- 10 malware analysis patterns
- Additional unified debugging pattern
- Additional user interface problem analysis pattern
- Additional pattern classification including memory and log acquisition patterns
- Additional .NET memory analysis patterns
- Introduction to software problem description patterns
- Introduction to software diagnostics patterns
- Introduction to general abnormal structure and behaviour patterns
- Introduction to software disruption patterns
- Introduction to static code analysis patterns
- Introduction to network trace analysis patterns
- Introduction to software diagnostics report schemes
- Introduction to elementary software diagnostics patterns
- Introduction to patterns of software diagnostics architecture
- Introduction to patterns of disassembly, reconstruction and reversing
- Introduction to vulnerability analysis patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, Volume 5, and Volume 6

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 7
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 523 pages
  • Publisher: OpenTask (June 2014)
  • ISBN-13: 978-1-908043-51-1
  • Hardcover: 523 pages
  • Publisher: OpenTask (June 2014)
  • ISBN-13: 978-1-908043-52-8

Table of Contents
Errata

Back cover features a "liquid memory" image created with Photoshop from contents of computer memory.

Vulnerability Analysis Patterns (VAP)

These are general patterns of software vulnerability: synthesis of analysis patterns from the following software diagnostics catalogues:

  • MAP - Memory Analysis Patterns (include behavioural and structural patterns)
  • TAP - Trace Analysis Patterns
  • CAP - Code Analysis Patterns (previously introduced as Static Code Analysis Patterns)
  • ADDR - Deconstruction, Disassembly, and Reversing (binary equivalent of CAP) Patterns (the current list is available here)
  • Also:

    VEC - Vulnerability, Exploit, and Control of Victimware
    Victimware - bugs of software (including bugs of malware) + vulnerabilities (provocative and precipitative victimware)

    For victimware classification please look at this presentation: http://www.patterndiagnostics.com/Victimware-materials

    The following easy to remember diagram combines all these acronyms and terminology:

    The first pattern we suggest is called Versioned Namespace. It is similar to Namespace malware analysis pattern but covers victimware side. Not only some API sets seen from source code and binaries but their versions also have importance. Again, this proposed new catalogue contains general analysis patterns, not specific operating system and product patterns. More patterns will be added later according to our pattern-based software diagnostics incremental and iterative methodology.

Software Diagnosis Codes

While working on Diagnostic Manual of Software Problems (DMS) announced last year we found the need to introduce software diagnostic codes. The proposed natural candidate schema is based on pattern orientation and pattern catalogues. It consists of a major and minor codes. The major code is a combination of one letter software artefact type, three letter structural classifier, and 3 letter behavioural pattern classifier:

Artefact Type - Structural Pattern - Behavioural Pattern

For example:

M-THR-SPK

Memory - Thread - Spike which corresponds to Spiking Thread pattern from memory analysis catalogue.

The optional minor code is not currently specified but may include pattern implementation such as operating system platform including CPU architecture, for, example: WIN.X32 or OSX.X64. So the final code may look like:

M-THR-SPK.WIN.X64

Software Diagnostics Metaphors

Software Diagnostics as Psychology

Analogy: studying how code construction ideas are execute.

  • Philosophy - Software Construction
  • Psychology - Software Diagnostics
  • Physiology - Software Execution

Software Diagnostics as Literary Criticism

Analogy: studying patterns across software execution artefacts such as software narratives (traces and logs) and memory snapshots.

  • Writing Fiction - Software Construction
  • Reading Fiction - Software Execution
  • Reviewing Fiction - Traditional Software Diagnostics
  • Literary Criticism - Pattern-Oriented Software Diagnostics

Rapid Software Diagnostics Process (RSDP)

Increased complexity of software incidents and their growing numbers require fast responses. We reviewed and partitioned our pattern catalogues into two tiers. The first tier requires less analysis efforts and provides faster response time than the second tier. The revised memory and trace analysis catalogues together with adjusted checklists will be published soon.

Right First Time Software Diagnosis

Right First Time Software Diagnostics is based on pattern-oriented diagnostics process and pattern catalogues.

Syndicate content