Archive for the ‘Root Cause Analysis’ Category
Monday, December 12th, 2016
Recently we found a correlation between software trace with high Statement Density and Current of Periodic Error with uniform Error Distribution and process heap Memory Leak suspected from memory dump analysis. If we metaphorically view periodic errors as “frequency” and the size of a heap as “mass” we may see that the growth of “frequency” correlates with the growth of “mass” and vice versa. Since frequency is inversely proportional to wave length we see a metaphorical analog to Louis de Broglie’s wave-particle duality. In general, as we already pointed in the discussion of narrativity and spatiality of software execution artifacts (see also Software Trace and Memory Dump Analysis seminar), software traces / logs and memory dumps can be seen as “dual” to each other according (metaphorically again) to de Broglie’s “duality of the laws of nature”. So we name this analysis pattern De Broglie Trace Duality since some memory dump regions can be considered of a general trace nature. Our correlation can be depicted in this diagram:

Practically, when we see Memory Consumption Patterns (but don’t know their source / root cause yet) we can ask for traces and logs, and in the case of frequent Periodic Errors found there we can suggest troubleshooting steps that may serve as a resolution or workaround.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Log Analysis, Root Cause Analysis, Software Trace Analysis, Trace Analysis Patterns, Trace Analysis and Physics | No Comments »
Monday, June 27th, 2016
Sometimes we have trace and log messages that appear with certain time frequency throughout all log or specific Thread of Activity. Such frequencies may fluctuate reflecting varying system or process performance. Analyzing trace areas where such messages have different Time Deltas may point to additional diagnostic log messages useful for root cause analysis. The following minimal trace graph depicts the recent log analysis for proprietary file copy operation where the frequency of internal communication channel entry/exit Opposition Messages was decreasing from time to time. Such periods were correlating with increased time intervals between “entry” and “exit” messages. Analysis of messages between them revealed additional diagnostic statements missing in periods of higher frequency and corresponding Timeouts adding up to overall performance degradation and slowness of copy operation.

Additional analysis of Data Association in a different message type between available communication buffers and the total number of such buffers revealed significant frequency drop during constant Data Flow of zero available communication buffers:

We call this analysis pattern Fourier Activity by analogy with Fourier series in mathematics. This pattern is for individual message types and can also be considered a fine-grained example of Statement Current and Trace Acceleration analysis patterns which can be used to detect areas of different frequencies in individual Fibers (Adjoint Threads of Activities formed from the same Thread of Activity).
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in CDF Analysis Tips and Tricks, Log Analysis, Root Cause Analysis, Software Trace Analysis, Trace Analysis Patterns | No Comments »
Sunday, September 23rd, 2012
Learn from this Webinar about phenomenological, hermeneutical and analytical approaches to software diagnostics and its knowledge, foundations, norms, theories, logic, methodology, language, ontology, nature and truth. This seminar is hosted by Software Diagnostics Services.

Title: Introduction to Philosophy of Software Diagnostics
Date: 17th of December, 2012
Time: 19:00 GMT
Duration: 60 minutes
Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/872846486
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Computer Science, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Education and Research, Escalation Engineering, First Fault Software Diagnostics, Hermeneutics of Memory Dumps and Traces, Logic, Memiotics (Memory Semiotics), Memoretics, Memory Dump Analysis Methodology, Performance Monitoring, Phenomenology of Software Diagnostics, Philosophy, Philosophy of Software Diagnostics, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Semiotics, Software Behavior Patterns, Software Diagnostics, Software Diagnostics Institute, Software Diagnostics Pattern Language, Software Diagnostics Patterns, Software Diagnostics Services, Software and Philosophy, Systems Theory, Systems Thinking, Testing, The Way of Philip Marlowe, Trace Analysis Patterns, Unified Software Diagnostics, Webinars | No Comments »
Friday, July 13th, 2012
For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:
The First Comprehensive Software Diagnostics Service
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Cloud Memory Dump Analysis, Complete Memory Dump Analysis, Core Dump Analysis, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Debugging, Debugging Bureau, Debugging Industry, Debugging Methodology, Debugging Today, Debugging Trends, Education, Education and Research, Escalation Engineering, Event Tracing for Windows (ETW), First Fault Software Diagnostics, Generative Debugging, JIT Crash Analysis, JIT Memory Space Analysis, Java Debugging, Kernel Development, Kernel Memory Dump Analysis, Linux Crash Corner, MFC Debugging, Mac Crash Corner, Mac OS X, Malware Analysis, Memoretics, Memory Analysis Forensics and Intelligence, Memory Analysis Report System, Memory Dump Analysis Methodology, Memory Dump Analysis Services, Minidump Analysis, New Debugging School, Pattern-Driven Debugging, Pattern-Driven Software Support, Performance Monitoring, Root Cause Analysis, SQL Debugging, Security, Software Debugging Services, Software Diagnostics, Software Diagnostics Institute, Software Diagnostics Services, Software Engineering, Software Problem Solving, Software Technical Support, Software Trace Analysis, Software Trace Analysis Report Environment (STARE), Tools, Training and Seminars, Troubleshooting Methodology, Unified Software Diagnostics, Windows 7, Windows 8, Windows Azure, Windows Mobile, Windows Server 2008, Windows System Administration, x64 Mac OS X, x64 Windows | No Comments »
Sunday, April 15th, 2012
After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.
- Title: Memory Dump Analysis Anthology, Volume 1
- Author: Dmitry Vostokov
- Publisher: OpenTask (Summer 2012)
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 800 pages
- ISBN-13: 978-1-908043-35-1
- Hardcover: 800 pages
- ISBN-13: 978-1-908043-36-8
The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Aesthetics of Memory Dumps, Announcements, AntiPatterns, Art, Assembly Language, Best Practices, Books, Bugchecks Depicted, C and C++, Complete Memory Dump Analysis, Computer Science, Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, Debugging, Debugging Methodology, Dr. Watson, Escalation Engineering, Fun with Crash Dumps, GDB for WinDbg Users, Hardware, Images of Computer Memory, Kernel Development, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Dump Analysis Methodology, Memory Space Art, Memory Space Music, Memory Visualization, Minidump Analysis, Multithreading, Pattern-Driven Debugging, Pattern-Driven Software Support, Publishing, Reference, Root Cause Analysis, Science of Memory Dump Analysis, Software Architecture, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Diagnostics, Software Engineering, Software Technical Support, Stack Trace Collection, Testing, Tools, Troubleshooting Methodology, Vista, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Data Structures, Windows Server 2008, Windows System Administration, x64 Windows | No Comments »
Sunday, April 15th, 2012
In a week this short full color book should appear in online bookstores:
- Title: Software Narratology: An Introduction to the Applied Science of Software Stories
- Authors: Dmitry Vostokov, Memory Dump Analysis Services
- Description: This is a transcript of Memory Dump Analysis Services Webinar about Software Narratology: an exciting new discipline and a field of research founded by DumpAnalysis.org. When software executes it gives us its stories in the form of UI events, software traces and logs. Such stories can be analyzed for their structure and patterns for troubleshooting, debugging and problem resolution purposes. Topics also include software narremes and their types, anticipatory software construction and software diagnostics.
- Publisher: OpenTask (April 2012)
- Language: English
- Product Dimensions: 28.0 x 21.6
- Paperback: 26 pages
- ISBN-13: 978-1908043078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Books, CDF Analysis Tips and Tricks, Code Reading, Computer Science, Debugging, Debugging Methodology, Debugging Trends, Education and Research, Escalation Engineering, Event Tracing for Windows (ETW), Hermeneutics of Memory Dumps and Traces, Ideas, Narralog Programming Language, New Debugging School, Pattern-Driven Debugging, Pattern-Driven Software Support, Presentations, Publishing, Root Cause Analysis, SPDL, Science of Software Tracing, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Diagnostics, Software Engineering, Software Narrative Fiction, Software Narrative Science, Software Narratology, Software Narremes, Software Problem Description Patterns, Software Problem Solving, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Linguistics, Software Trace Modeling, Software Trace Reading, Software Trace Visualization, Software Tracing Implementation Patterns, Software Troubleshooting Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Unified Software Diagnostics, Webinars | No Comments »
Monday, April 9th, 2012
Memory Dump Analysis Services organizes a free Webinar on Unified Software Diagnostics (USD) and the new scalable cost-effective software support model called Pattern-Driven Software Support devised to address various shortcomings in existing tiered software support organizations. Examples cover Windows, Mac OS and Linux.

Date: 22nd of June, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes
Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/172771078
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Best Practices, Crash Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Debugging Trends, Economics, Escalation Engineering, Event Tracing for Windows (ETW), JIT Crash Analysis, JIT Memory Space Analysis, Malware Analysis, Malware Patterns, Memoretics, Memory Analysis Forensics and Intelligence, Memory Dump Analysis Methodology, Memory Dump Analysis Services, New Acronyms, New Debugging School, Pattern Prediction, Pattern-Driven Debugging, Pattern-Driven Software Support, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Diagnostics, Software Engineering, Software Narratology, Software Problem Description Patterns, Software Problem Solving, Software Support Patterns, Software Technical Support, Software Trace Analysis, Software Troubleshooting Patterns, Software Victimology, Software and Economics, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Unified Software Diagnostics, Victimware, Webinars, Workaround Patterns | No Comments »
Monday, February 20th, 2012
I was recently asked by a group of trainees to outline a simple approach to proceed after opening a memory dump. So I came up with these 7 steps:
1. !analyze -v [-hang]
2. Exception (Bugcheck): stack trace analysis with d* and lmv
3. !locks
4. !runaway f (!running)
5. Dump all (processes and) thread stack traces [with 32-bit] ~*kv (!process 0 ff)
6. Search for signs/patterns of abnormal behavior (exceptions, wait chains, message boxes [, from your custom checklist])
7. Narrow analysis down to a specific thread and dump raw stack data if needed [repeat*]
(commands/options in brackets denote kernel/complete dump variation)
[notes in square brackets denote additional options, such as x64 specifics, your product details, etc.]
What are your steps? I would be interested to hear about alternative analysis steps, techniques, etc.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Best Practices, Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Memory Dump Analysis Methodology, Root Cause Analysis, Software Trace Analysis, WinDbg Tips and Tricks, x64 Windows | 1 Comment »
Thursday, November 3rd, 2011
The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:
- 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
- 4 new pattern interaction case studies
- 11 new trace analysis patterns
- New Debugware pattern
- Introduction to UI problem analysis patterns
- Introduction to intelligence analysis patterns
- Introduction to unified debugging pattern language
- Introduction to generative debugging, metadefect template library and DNA of software behaviour
- The new school of debugging and trends
- .NET memory dump analysis checklist
- Software trace analysis checklist
- Introduction to close and deconstructive readings of a software trace
- Memory dump analysis compass
- Computical and Stack Trace Art
- The abductive reasoning of Philip Marlowe
- Orbifold memory space and cloud computing
- Memory worldview
- Interpretation of cyberspace
- Relationship of memory dumps to religion
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5
Product information:
- Title: Memory Dump Analysis Anthology, Volume 6
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 300 pages
- Publisher: Opentask (December 2011)
- ISBN-13: 978-1-908043-19-1
- Hardcover: 300 pages
- Publisher: Opentask (January 2012)
- ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Announcements, Art, Books, Cloud Computing, Cloud Memory Dump Analysis, Common Mistakes, Complete Memory Dump Analysis, Computer Science, Computicart (Computical Art), Crash Dump Analysis, Crash Dump Patterns, Cyber Intelligence, Cyber Problems, Cyber Security, Cyber Space, Cyber Warfare, DebugWare Patterns, Debugging, Debugging Industry, Debugging Methodology, Debugging Slang, Debugging Trends, Escalation Engineering, Generative Debugging, Intelligence Analysis Patterns, Kernel Development, Memoidealism, Memoretics, Memory Visualization, Metadefect Template Library, New Debugging School, Philosophy, Physicalist Art, Publishing, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Deconstruction, Software Trace Reading, Software Victimology, Testing, The Way of Philip Marlowe, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Victimware, WinDbg Tips and Tricks, Windows 7, Windows Azure, Windows Data Structures, Windows Server 2008, Windows System Administration, x64 Windows | No Comments »
Wednesday, September 21st, 2011
Memoretics promotes pattern-driven memory dump and software trace analysis which has many uses but not limited to:
- Software and site reliability
- Software Debugging
- QA and Software Testing
- Computer Security
- Software Troubleshooting
- Malware Research and Analysis
- Tools as a Service (TaaS)
- Supportability
- Software Diagnostics

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Cloud Memory Dump Analysis, Computer Forensics, Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Trends, Escalation Engineering, Malware Analysis, Malware Patterns, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Metadefect Template Library, Metamalware, Pattern Models, Pattern Prediction, Reverse Engineering, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Defect Construction, Software Engineering, Software Generalist, Software Narratology, Software Support Patterns, Software Technical Support, Software Trace Analysis, Software Troubleshooting Patterns, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Victimware, Windows System Administration, Workaround Patterns | No Comments »
Wednesday, September 21st, 2011
Memoretics as a science of memory snapshots borrows many ideas from the following disciplines (the list is not exhaustive):
- Troubleshooting and Debugging
- Intelligence Analysis
- Critical Thinking
- Forensics
- Linguistics
- Archaeology
- Psychoanalysis
- History
- Mathematics: Sets and Categories
- Literary Criticism and Narratology
It also contributes many ideas back. The following diagram depicts such an interaction:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Archaeology of Computer Memory, Categorical Debugging, Computer Forensics, Computer Science, Countefactual Debugging, Crash Dump Analysis, Crash Dump De-analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Dublin School of Security, Education and Research, Escalation Engineering, General Memory Analysis, Generative Debugging, Hermeneutics of Memory Dumps and Traces, History, Intelligence Analysis Patterns, M->analysis, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Memory Dump Analysis and History, Memory Systems Language, Memory Visualization, Metadefect Template Library, Metamalware, New Debugging School, Paleo-debugging, Pattern Models, Pattern Prediction, Psychoanalysis of Software Maintenance and Support, Psychology, Reverse Engineering, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Linguistics, Software Trace Reading, Software Trace Visualization, Software Tracing Implementation Patterns, Software Troubleshooting Patterns, Software Victimology, Software and History, Structural Memory Analysis and Social Sciences, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Victimware, Virtualization, Visual Dump Analysis, Workaround Patterns | No Comments »
Sunday, August 14th, 2011
Due to the need to extend existing basic and intermediate Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises advanced training course. Here is the description and registration information:
Learn how to navigate through memory dump space and Windows data structures to troubleshoot and debug complex software incidents. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose structural and behavioral patterns in 32-bit and 64-bit process, kernel and complete memory dumps.

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.
The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:
- A full transcript in PDF format (retail price $200)
- 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
- A personalized attendance certificate with unique CID (PDF format)
Prerequisites: Basic and intermediate level Windows memory dump analysis: ability to list processors, processes, threads, modules, apply symbols, walk through stack traces and raw stack data, diagnose patterns such as heap corruption, CPU spike, memory and handle leaks, access violation, stack overflow, critical section and resource wait chains and deadlocks. If you don’t feel comfortable with prerequisites then Accelerated Windows Memory Dump Analysis training is recommended to take (or purchase a corresponding book) before attending this course.
Audience: Software developers, software technical support and escalation engineers.
Session: December 9, 2011 4:00 PM - 8:00 PM GMT
Price: 210 USD
Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/3788047691824598784
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Escalation Engineering, Kernel Development, Memory Dump Analysis Services, Multithreading, Reverse Engineering, Root Cause Analysis, Software Engineering, Software Technical Support, Structural Memory Patterns, Training and Seminars, Uses of UML, Vista, WinDbg Scripts, WinDbg Tips and Tricks, Windows 7, Windows Data Structures, Windows Server 2008, x64 Windows | No Comments »
Sunday, May 15th, 2011
Presentation Software Trace and Memory Dump Analysis: Patterns, Tools, Processes and Best Practices from E2E Virtualization Conference (13th of May, 2011) is available for download:
http://www.dumpanalysis.com/STMDA-materials
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Debugging, Debugging Methodology, Escalation Engineering, Presentations, Root Cause Analysis, Software Behavior Patterns, Software Technical Support, Software Trace Analysis, Software Trace Reading, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology | No Comments »
Friday, April 22nd, 2011
Very simple methodology summarized in just 3 words applicable to any problem solving domain and its activities including software troubleshooting and debugging:
Artifacts. Checklists. Patterns.
As an example of checklists and patterns please see these 3 presentations related to memory dump and software trace artifacts:
Introduction to Pattern-Driven Software Problem Solving
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Pattern-Driven Memory Dump Analysis
I’m now working on a detailed presentation of this methodology.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Debugging, Debugging Methodology, Escalation Engineering, New Acronyms, Root Cause Analysis, Software Technical Support, Troubleshooting Methodology | No Comments »