Software Diagnostics Library

We resume our workaround patterns for common reusable solutions to common software execution problems. In the past we proposed a general pattern of Axed Code for removing problem software behavior. The Shadow File blog post about fixing free() crashes by introducing heap metadata header with a data length set to zero led me to generalize and introduce a complementary Axed Data pattern. Such a pattern suggests cutting the data size specified in metadata memory plane (which may be separate from the data plane). In some cases, it may avoid buffer overwrites including Local and Shared in addition to Dynamic Memory Corruption (process heap and kernel pool). The following picture illustrates this general pattern approach.

Sometimes, it may even be possible to provide a workaround by cutting real file data, for example, by changing its file or database record size. But this is also done conceptually by changing file or database system metadata.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services organizes a free Webinar on Unified Software Diagnostics (USD) and the new scalable cost-effective software support model called Pattern-Driven Software Support devised to address various shortcomings in existing tiered software support organizations. Examples cover Windows, Mac OS  and Linux.

Date: 22nd of June, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/172771078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes Problem Module pattern can help in troubleshooting. Problem modules (including process names) are components that due to their value adding behaviour might break normal software behaviour and therefore require some troubleshooting workarounds from minor configuration changes to complete removal. Typical examples include memory optimization services for terminal services environments or hooksware. Typically you can see main process modules in the output of !vm or !process 0 0 commands. lm command will list module names such as DLLs from a process memory dump, lmk command can give you the list of kernel space modules (for example, drivers) from kernel and complete memory dumps, and the following command lists all user space modules for each process in a complete memory dump:

!for_each_process ".process /r /p @#Process; lmu"

Of course you can also try various lm command variants if you are interested in timestamps and module information.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoretics promotes pattern-driven memory dump and software trace analysis which has many uses but not limited to:

• Software and site reliability
• Software Debugging
• QA and Software Testing
• Computer Security
• Software Troubleshooting
• Malware Research and Analysis
• Tools as a Service (TaaS)
• Supportability
• Software Diagnostics

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoretics as a science of memory snapshots borrows many ideas from the following disciplines (the list is not exhaustive):

• Troubleshooting and Debugging
• Intelligence Analysis
• Critical Thinking
• Forensics
• Linguistics
• Archaeology
• Psychoanalysis
• History
• Mathematics: Sets and Categories
• Literary Criticism and Narratology

It also contributes many ideas back. The following diagram depicts such an interaction:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes Fake API calls can be an alternative to source code defect fixing especially if it provides quick problem resolution in production environment. Such faking can be implemented at binary level (as patching and hooking after link phase) or at object code level (before link phase). For the latter example please look at the following case study: Applying API Wrapper Pattern. The reverse process (introducing unexpected situations via unit testing, modeling software defects) is similar and for C and C++ can be done using Isolator++.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 5 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+5

It features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Its table of contents is available here:

www.dumpanalysis.org/MDAA/MDA-Anthology-V5-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 6 that should be available in November-December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The presentation materials from the webinar (25th of March, 2011) are available for download:

http://www.dumpanalysis.com/PDSPSI-materials

Thanks to everyone who registered and attended!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first Webinar to start an in-depth discussion of pattern-driven software troubleshooting, debugging and maintenance:

Date: 25th of March 2011
Time: 18:30 (GMT) 14:30 (EST) 11:30 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/448268158

Topics include:

• A Short History of DumpAnalysis.org
• Memory Dump Analysis Patterns
• Troubleshooting and Debugging Tools (Debugware) Patterns
• Software Trace Analysis Patterns
• From Software Defects to Software Behavior
• Workaround Patterns
• Structural Memory Patterns
• Memory Analysis Domain Pattern Hierarchy
• New Directions

Prerequisites: experience in software troubleshooting and/or debugging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

With the new year starts the new initiative to integrate traditional multidisciplinary debugging approaches and methodologies with multiplatform pattern-driven software problem solving, unified debugging patterns, best practices in memory dump analysis and software tracing, computer security, economics, and the new emerging trends I’m going to write about during this year.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As the new decade is approaching (2011-2020) we would like to make a few previews and predictions:

- Increased complexity of software will bring more methods from biological, social sciences and humanities in addition to existing methods of automated debugging and computer science techniques

- Focus on first fault software problem solving (when aspect)

- Focus on pattern-driven software problem solving (how aspect)

- Fusion of debugging and malware analysis into a unified structural and behavioral pattern framework

- Visual debugging, memory and software trace visualization techniques

- Software maintenance certification

- Focus on domain-driven troubleshooting and debugging tools as a service (debugware TaaS)

- Focus on security issues related to memory dumps and software traces

- New scripting languages and programming language extensions for debugging

- The maturation of the science of memory snapshots and software traces (memoretics)

Imagining is not not limited to the above and more to come and explain in the forthcoming parts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services announces the launch of the first Memory Dump Analysis Audit Service:

http://www.dumpanalysis.com/memory-dump-analysis-audit-service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca

The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Product information:

• Title: Memory Dump Analysis Anthology, Volume 5
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-96-4

• Hardcover: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 4 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4

It features:

- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art

Its table of contents is available here:

http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):

• Are you interested in on-site training, prefer traveling or attending webinars?
• Are you interested in software trace analysis as well?
• What specific topics are you interested in?
• What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)

Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.

Looking forward to your responses. Any suggestions are welcome.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

What happens when Hidden Output and Frozen Process patterns don’t help with annoying popup windows? The former can’t prevent windows from reappearing afresh and the latter could block other coupled processes that might exchange window messages with our suspended process or simply use any IPC mechanism. Here Axed Code pattern can help as demonstrated below. One process was frequently and briefly showing network disconnection message box or dialog. The problem is that it was also bringing its main window into foreground disrupting work in other windows because they were loosing focus. Next time the dialog appeared we found its process ID in Task Manager and attached WinDbg to it. We wasn’t sure what dialog function to intercept so we put a general breakpoint on all “Dialog” functions for all threads:

0:000:x86> bm *Dialog*
[...]
  6: 73a8ba81 @!"MFC80!CDialog::~CDialog"
  7: 73ac25e2 @!"MFC80!CPageSetupDialog::~CPageSetupDialog"
  8: 73a94b6b @!"MFC80!CDHtmlDialog::_AfxSimpleScanf"
  9: 73a8fbe9 @!"MFC80!CFileDialog::OnTypeChange"
 10: 73a90b17 @!"MFC80!CColorDialog::GetRuntimeClass"
 11: 73a8bb4a @!"MFC80!CDialog::CreateIndirect"
[...]
360: 73a93750 @!"MFC80!CDHtmlDialog::OnNavigateComplete"
361: 73a8f1f3 @!"MFC80!CCommonDialog::OnOK"
362: 73a95d9f @!"MFC80!CDHtmlDialog::GetDropTarget"
363: 73a90266 @!"MFC80!CPrintDialog::GetDevMode"
364: 73ac1514 @!"MFC80!COleInsertDialog::COleInsertDialog"
365: 73ac27c7 @!"MFC80!COlePropertiesDialog::COlePropertiesDialog"
366: 73a75282 @!"MFC80!CWnd::UpdateDialogControls"
367: 73a7fd86 @!"MFC80!CDialogBar::SetOccDialogInfo"

0:000:x86> g
Breakpoint 314 hit
MFC80!_AfxPostInitDialog:
73a7134e 55              push    ebp

0:000:x86> kL 100
ChildEBP RetAddr  Args to Child             
0027ed2c 73a7180a MFC80!_AfxPostInitDialog
0027ed90 75628817 MFC80!_AfxActivationWndProc+0x90
0027edbc 7562898e USER32!InternalCallWinProc+0x23
0027ee34 7562c306 USER32!UserCallWinProcCheckWow+0x109
0027ee78 756375a2 USER32!SendMessageWorker+0x55b
0027ef4c 7563787a USER32!InternalCreateDialog+0xb64
0027ef70 75649b65 USER32!CreateDialogIndirectParamAorW+0x33
0027ef9c 75225192 USER32!CreateDialogParamA+0x4a
WARNING: Stack unwind information not available. Following frames may be wrong.
0027efc8 010c3bf1 DllA!WarningPopup+0×152
0027effc 73a71812 ProcessA+0×9fa1
00000000 00000000 MFC80!_AfxActivationWndProc+0×98

Now we cleared all breakpoints and put the new breakpoint on WarningPopup function:

0:000:x86> bc *

0:000:x86> bp DllA!WarningPopup

0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 51              push    ecx

Then we assumed that the calling convention was the default one used by C or C++ code like _cdecl and took the bold step to replace push ecx with ret instruction:

0:000:x86> a 75225040
75225040 ret
ret
75225041

0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 c3 ret

0:000:x86> bc *

0:000:x86> g

Result: no warning popups anymore.

I originally intended to name the pattern Patched Code but then realized that code axing can also be done at the source code level as a quick temporal fix.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Another workaround pattern for some problems is to freeze a process responsible for an annoying or excessive activity like in the case study: Debugger as a Shut Up Application. We can also use other tools for this purpose like Mark Russinovich’s PsSuspend. The suitable name for this pattern is Frozen Process.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After fighting HTML comments in Safari and Chrome (see the case study below) I came to an idea to name and catalog workaround patterns in troubleshooting and debugging. The first one is called Hidden Output. Sometimes we can just remove message boxes reporting minor problems and generating unnecessary support calls by hiding their windows, for example, by using CtxHideEx32. A different example is what I did today when troubleshooting Amazon aStore widget HTML code. It worked well in IE8:

However, in Apple Safari and Google Chrome the widget code was visible at the top of the page:

After a few unsuccessful attempts to debug the problem and faced with other pressing tasks I got a flash in my mind to hide the visible code by changing its color to be the same as its background:

<font color=”D3E7F4″><script type=”text/javascript”><!–
amazon_ad_tag=”crasdumpanala-20″;
amazon_ad_width=”728″;
amazon_ad_height=”90″;
amazon_color_background=”D3E7F4″;
amazon_color_border=”0000FF”;
amazon_color_logo=”FFFFFF”;
amazon_color_link=”0000FF”;
amazon_ad_logo=”hide”;
amazon_ad_link_target=”new”;
amazon_ad_border=”hide”;
amazon_ad_title=”OpenTask Books, Magazines and Notebooks”; //–></script>
<script type=”text/javascript” src=”http://www.assoc-amazon.com/s/asw.js”></script></font>
 
After that the picture became nicer:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -