Archive for the ‘Software Technical Support’ Category
Monday, March 15th, 2010
On The Same Page - coming to the same conclusion as another engineer when looking at a memory dump or a software trace. Literally means the same page of memory where an exception occurred or a stack trace is reconstructed or the same “page” when browsing a software trace output using a viewer.
Examples: Aha, we are on the same page!
With fix-privet,
Dr. DebugLove
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Debugging, Debugging Slang, Escalation Engineering, Fun with Crash Dumps, Software Technical Support, Software Trace Analysis | No Comments »
Thursday, February 11th, 2010
This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:
- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes
Product information:
- Title: Memory Dump Analysis Anthology, Volume 4
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 410 pages
- Publisher: Opentask (30 March 2010)
- ISBN-13: 978-1-906717-86-5
- Hardcover: 410 pages
- Publisher: Opentask (30 April 2010)
- ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Aesthetics of Memory Dumps, Announcements, AntiPatterns, Art, Assembly Language, Books, C and C++, CDA Pattern Classification, CDF Analysis Tips and Tricks, Categorical Debugging, Certification, Citrix, Code Reading, Common Mistakes, Computer Science, Countefactual Debugging, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, DebugWare Patterns, Debugging, Debugging Slang, Education and Research, Escalation Engineering, Fun with Crash Dumps, Games for Debugging, Hardware, Horrors of Computation, Hyper-V, Kernel Development, Laws of Troubleshooting and Debugging, Mathematics of Debugging, Memiotics, Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Dreams, Memory Dump Analysis Jobs, Memory Dumps in Movies, Memory Space Art, Memory Visualization, Memuonics, Minidump Analysis, Multithreading, Music for Debugging, New Acronyms, New Words, Opcodism, Philosophy, Physicalist Art, Publishing, Science Fiction, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Visualization, Software Tracing for Dummies, Software Victimology, Stack Trace Collection, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Victimware, Virtualization, Vista, Visual Dump Analysis, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns | No Comments »
Friday, February 5th, 2010
STUPID - Software Tracing Under Pressure Is Dumb
Examples: STUPID! STUPID! I told you to enable all modules! You included all but the one I need…
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Debugging, Debugging Slang, New Acronyms, Software Technical Support, Software Trace Analysis | No Comments »
Tuesday, January 26th, 2010
What happens when Hidden Output and Frozen Process patterns don’t help with annoying popup windows? The former can’t prevent windows from reappearing afresh and the latter could block other coupled processes that might exchange window messages with our suspended process or simply use any IPC mechanism. Here Axed Code pattern can help as demonstrated below. One process was frequently and briefly showing network disconnection message box or dialog. The problem is that it was also bringing its main window into foreground disrupting work in other windows because they were loosing focus. Next time the dialog appeared we found its process ID in Task Manager and attached WinDbg to it. We wasn’t sure what dialog function to intercept so we put a general breakpoint on all “Dialog” functions for all threads:
0:000:x86> bm *Dialog*
[...]
6: 73a8ba81 @!"MFC80!CDialog::~CDialog"
7: 73ac25e2 @!"MFC80!CPageSetupDialog::~CPageSetupDialog"
8: 73a94b6b @!"MFC80!CDHtmlDialog::_AfxSimpleScanf"
9: 73a8fbe9 @!"MFC80!CFileDialog::OnTypeChange"
10: 73a90b17 @!"MFC80!CColorDialog::GetRuntimeClass"
11: 73a8bb4a @!"MFC80!CDialog::CreateIndirect"
[...]
360: 73a93750 @!"MFC80!CDHtmlDialog::OnNavigateComplete"
361: 73a8f1f3 @!"MFC80!CCommonDialog::OnOK"
362: 73a95d9f @!"MFC80!CDHtmlDialog::GetDropTarget"
363: 73a90266 @!"MFC80!CPrintDialog::GetDevMode"
364: 73ac1514 @!"MFC80!COleInsertDialog::COleInsertDialog"
365: 73ac27c7 @!"MFC80!COlePropertiesDialog::COlePropertiesDialog"
366: 73a75282 @!"MFC80!CWnd::UpdateDialogControls"
367: 73a7fd86 @!"MFC80!CDialogBar::SetOccDialogInfo"
0:000:x86> g
Breakpoint 314 hit
MFC80!_AfxPostInitDialog:
73a7134e 55 push ebp
0:000:x86> kL 100
ChildEBP RetAddr Args to Child
0027ed2c 73a7180a MFC80!_AfxPostInitDialog
0027ed90 75628817 MFC80!_AfxActivationWndProc+0x90
0027edbc 7562898e USER32!InternalCallWinProc+0x23
0027ee34 7562c306 USER32!UserCallWinProcCheckWow+0x109
0027ee78 756375a2 USER32!SendMessageWorker+0x55b
0027ef4c 7563787a USER32!InternalCreateDialog+0xb64
0027ef70 75649b65 USER32!CreateDialogIndirectParamAorW+0x33
0027ef9c 75225192 USER32!CreateDialogParamA+0x4a
WARNING: Stack unwind information not available. Following frames may be wrong.
0027efc8 010c3bf1 DllA!WarningPopup+0×152
0027effc 73a71812 ProcessA+0×9fa1
00000000 00000000 MFC80!_AfxActivationWndProc+0×98
Now we cleared all breakpoints and put the new breakpoint on WarningPopup function:
0:000:x86> bc *
0:000:x86> bp DllA!WarningPopup
0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 51 push ecx
Then we assumed that the calling convention was the default one used by C or C++ code like _cdecl and took the bold step to replace push ecx with ret instruction:
0:000:x86> a 75225040
75225040 ret
ret
75225041
0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 c3 ret
0:000:x86> bc *
0:000:x86> g
Result: no warning popups anymore.
I originally intended to name the pattern Patched Code but then realized that code axing can also be done at the source code level as a quick temporal fix.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Debugging, Software Technical Support, Troubleshooting Methodology, WinDbg Tips and Tricks, Workaround Patterns | No Comments »
Monday, January 25th, 2010
Another workaround pattern for some problems is to freeze a process responsible for an annoying or excessive activity like in the case study: Debugger as a Shut Up Application. We can also use other tools for this purpose like Mark Russinovich’s PsSuspend. The suitable name for this pattern is Frozen Process.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Debugging, Software Technical Support, Tools, Troubleshooting Methodology, Workaround Patterns | 1 Comment »
Sunday, January 24th, 2010
After fighting HTML comments in Safari and Chrome (see the case study below) I came to an idea to name and catalog workaround patterns in troubleshooting and debugging. The first one is called Hidden Output. Sometimes we can just remove message boxes reporting minor problems and generating unnecessary support calls by hiding their windows, for example, by using CtxHideEx32. A different example is what I did today when troubleshooting Amazon aStore widget HTML code. It worked well in IE8:

However, in Apple Safari and Google Chrome the widget code was visible at the top of the page:
After a few unsuccessful attempts to debug the problem and faced with other pressing tasks I got a flash in my mind to hide the visible code by changing its color to be the same as its background:
<font color=”D3E7F4″><script type=”text/javascript”><!–
amazon_ad_tag=”crasdumpanala-20″;
amazon_ad_width=”728″;
amazon_ad_height=”90″;
amazon_color_background=”D3E7F4″;
amazon_color_border=”0000FF”;
amazon_color_logo=”FFFFFF”;
amazon_color_link=”0000FF”;
amazon_ad_logo=”hide”;
amazon_ad_link_target=”new”;
amazon_ad_border=”hide”;
amazon_ad_title=”OpenTask Books, Magazines and Notebooks”; //–></script>
<script type=”text/javascript” src=”http://www.assoc-amazon.com/s/asw.js”></script></font>
After that the picture became nicer:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Citrix, Debugging, Software Technical Support, Tools, Troubleshooting Methodology, Workaround Patterns | 1 Comment »
Monday, January 18th, 2010
After exciting results of the previous year of debugging it is time to announce modest plans for this year, 0×7DA:
Release the first beta version of EasyDbg
Release the first beta version of CARE (Crash Analysis Report Environment) for a pattern-driven debugger log analyzer with standards for structured audience-driven reports
Release the first beta version of STARE (Software Trace Analysis Report Environment) for a pattern-driven software trace analyzer with corresponding standards for structured audience-driven reports
Publish the following books on dump analysis that address different audiences (general users, system administrators, support and escalation engineers, testers, software engineers, security and software defect researchers):
- Windows Debugging Notebook
- Crash Dump Analysis for System Administrators and Support Engineers
- Memory Dump Analysis Anthology, Volume 4
- Memory Dump Analysis Anthology, Volume 5
- Memory Dump Analysis Anthology Color Supplement
- Principles of Memory Dump Analysis
- My Computer Crashes and Freezes: A Non-technical Guide to Software and Hardware Errors
- Linux, FreeBSD and Mac OS X Debugging: Practical Foundations
- Encyclopedia of Crash Dump Analysis Patterns
- WinDbg In Use: Debugging Exercises
Publish articles related to memory dump analysis in Debugged! magazine
Update WinDbg Poster and Cards
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Books, Certification, Crash Dump Analysis, Debugging, Education and Research, Escalation Engineering, FreeBSD Crash Corner, Linux Crash Corner, Mac Crash Corner, New Acronyms, Publishing, Software Engineering, Software Technical Support, Software Trace Analysis | No Comments »
Thursday, January 14th, 2010
The Year of Debugging, 0×7D9, was a remarkable year for DumpAnalysis.org. Here is the list of achievements to report:
- Software Trace Analysis as a new discipline with its own set of patterns
- Unification of Memory Dump Analysis with Software Trace Analysis (DA+TA)
- New computer memory dump-based art movements: Opcodism and Physicalist Art
- Discovery of 3D computer memory visualization techniques
- Establishing Software Maintenance Institute
- Broadening software fault injection as Software Defect Construction discipline
- Establishing a new profession of a Software Defect Researcher
- Starting ambitious Dictionary of Debugging
- Publishing Windows Debugging: Practical Foundations book
- Publishing the first x86-free Windows debugging book: x64 Windows Debugging: Practical Foundations
- Establishing the new debugging magazine: Debugged! MZ/PE
- Publishing Memory Dump Analysis Anthology, Volume 3
- Cooperation with OpenTask to promote First Fault Software Problem Solving book
- Establishing Debugging Expert(s) Magazine Online
- Creating the first development process for debugging and software troubleshooting tools: RADII
- Publishing the first pattern-driven memory dump analysis troubleshooting methodology as a foundation for software debugging
- Proposal for an International Memory Analysts and Debuggers Day
- Almost completed Windows Debugging Notebook to be published soon
- The founder of DumpAnalysis.org (Dr. DebugLove) becomes a member of Citrix Systems Tweetrix Support Team
Now DumpAnalysis.org focuses on The Year of Dump Analysis, 0×7DA, as a foundation for the forthcoming debugging decade and reveals future plans this weekend.
I’m sure that many other organizations and individuals have no less remarkable accomplishments to report for 2009. I promise to track down and write about some of them in the forthcoming book:
The Science of Dr. Watson: An Illustrated History of Debugging (ISBN: 978-1906717070)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Art, Books, Citrix, Crash Dump Analysis, Debugged! MZ/PE, Debugging, Dictionary of Debugging, History, Memory Visualization, Publishing, Software Defect Construction, Software Engineering, Software Maintenance Institute, Software Technical Support, Software Trace Analysis, Trace Analysis Patterns | 1 Comment »
Sunday, December 27th, 2009
Main topics of Debugged! MZ/PE magazine issues for June and September 2010:
Debugged! MZ/PE: Multiplatform Software Defects, June, 2010 (Paperback, ISBN: 978-1906717902)
Debugged! MZ/PE: Software Defect Visualization and Annotation, September, 2010 (Paperback, ISBN: 978-1906717919)
Here are draft front covers designed today:


- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Books, Computer Science, Crash Dump Analysis, Debugged! MZ/PE, Debugging, Memory Visualization, Publishing, Software Engineering, Software Technical Support, Software Trace Analysis, Software Trace Visualization | No Comments »
Wednesday, December 23rd, 2009
See the greeting card on the portal together with New Year’s Eve code analysis puzzle:
DumpAnalysis.org Wishes Happy New Year 7DA!
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Assembly Language, Code Reading, Crash Dump Analysis, Debugging, Fun with Crash Dumps, Games for Debugging, Software Engineering, Software Technical Support | No Comments »
Sunday, December 20th, 2009
“Memory dumps are facts.”
I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:
Memory Dump Analysis Anthology, Volume 3
Table of Contents
In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.
The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4.
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in .NET Debugging, Aesthetics of Memory Dumps, Announcements, AntiPatterns, Art, Assembly Language, Books, Bugtations, C and C++, CDA Pattern Classification, CDF Analysis Tips and Tricks, Certification, Citrix, Code Reading, Common Mistakes, Computer Science, Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, DebugWare Patterns, Debugging, Debugging Slang, Deep Down C++, Education and Research, Escalation Engineering, Fun with Crash Dumps, Hardware, Horrors of Computation, Kernel Development, Mathematics of Debugging, Memiotics, Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Space Art, Memory Visualization, Minidump Analysis, Multithreading, Philosophy, Poetry, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Defect Construction, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Visualization, Software Tracing for Dummies, Software Victimology, Stack Trace Collection, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Victimware, Virtualization, Vista, Visual Dump Analysis, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration | 1 Comment »
Friday, December 18th, 2009
I propose to celebrate it on 08.08 every year starting from The Year of Dump Analysis, 2010, 7DA at 8:00 (I prefer 8pm for MAD Day, moderation is important in debugging too).
What do you think? If you count things from 0 or favour user space there is an alternative date: 07.07
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Crash Dump Analysis, Debugging, Software Engineering, Software Technical Support | 1 Comment »
Thursday, December 17th, 2009
Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday. The SMI web site temporary points to Crash Dump Analysis and Debugging Portal: www.sminstitute.com
Here is the current component structure of various institutions (depicted in UML):
Interface Tags:
IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Art, Certification, Computer Science, Crash Dump Analysis, Debugging, Education and Research, Escalation Engineering, History, Memiotics, Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Auralization, Memory Visualization, Philosophy, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Engineering, Software Maintenance Institute, Software Technical Support, Software Trace Analysis, Software Victimology, Testing, Training and Seminars | No Comments »
Wednesday, December 16th, 2009
Finally, after the long delay, the issue is available in print on Amazon and through other sellers:
Debugged! MZ/PE: Software Tracing


- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Books, CDF Analysis Tips and Tricks, Computer Science, Crash Dump Analysis, Crash Dump Patterns, Debugged! MZ/PE, Debugging, Education and Research, Fun with Crash Dumps, Memoretics, Memory Visualization, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Software Technical Support, Software Trace Analysis, Software Trace Visualization, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Visual Dump Analysis | 1 Comment »
Wednesday, December 9th, 2009
In its simplest form the first law of troubleshooting and debugging states that:
The more frequent a problem is, the simpler tool is needed to resolve and fix it.
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Bugtations, DebugWare Patterns, Debugging, Laws of Troubleshooting and Debugging, Software Technical Support, Tools, Troubleshooting Methodology | No Comments »
Wednesday, December 9th, 2009
I’m very pleased to announce that Dan Skwire’s unique book has been published by OpenTask:
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in Announcements, Books, First Fault Problem Solving, Publishing, Software Engineering, Software Technical Support, Tools, Troubleshooting Methodology | No Comments »
Tuesday, November 10th, 2009
Posted in Announcements, Crash Dump Analysis, Crash Dump Patterns, Debugged! MZ/PE, Debugging, Software Defect Construction, Software Technical Support, Software Trace Analysis, Trace Analysis Patterns | No Comments »
Friday, October 30th, 2009
Suppose we want to write a support tool to troubleshoot a frequent software problem. Would we start with GUI? Most tools are command line tools and we therefore choose Tool Façade pattern to decouple GUI from CLI:

By making our GUI optional we postpone its development until later implementation phase when we want to beautify or simplify the tool user interface. The troubleshooting of software problem we want to automate is quite complex, scattered across several knowledge base articles, with many troubleshooting steps and various checks. We therefore treat different steps as TUWs, Troubleshooting Units of Work (aggregating related ones if necessary) and choose Checklist pattern to coordinate them:

Initially we think that just one checklist is suffice but while studying the domain of troubleshooting we find that the problem usually spans client and server computers, where servers sometimes have client roles as well and different troubleshooting steps and checks are needed. In order to simplify the tool we apply Span Differentiator pattern and our architectural UML component diagram becomes more refined:

We also see that different TUW use similar operations like querying registry values or checking whether certain processes are running and factor them out into separate SMs, Supporting Modules:

Deployment methods are important in troubleshooting where it is not always desirable to change configuration on production servers and we plan for optional Self Extractor pattern:

In diagrams we liberally mixed component and static UML diagramming elements. We also need to design a uniform interface that each TUW module exposes to the main tool coordinator module but we leave it for the part 2.
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in DebugWare Patterns, Debugging, Software Architecture, Software Engineering, Software Technical Support | No Comments »
Wednesday, October 28th, 2009
Support tools can be packaged for delivery as (not counting optional separate readme and EULA files):
- - A raw collection of modules
- - A single installation package
- - A single archived file
- - A single self-extractable component
The latter common packaging choice is a pattern itself that I call Self Extractor. Typical examples include sysinternals tools like Process Monitor. Here is a simple unorthodox UML diagram for a single platform tool combining elements from static and component diagrams to illustrate component embedding (not sharing):

SM is a supporting module, for example, a driver, to carry out some task or receive notifications.
Some tools even extract different components depending on the processor architecture, for example, x86 and x64. This is illustrated on a more elaborate UML diagram:


The typical example would be a backward compatible x86 Extractor that extracts an x64 driver (Extractee) for an x64 OS kernel that doesn’t load x86 drivers.
- Dmitry Vostokov @ DumpAnalysis.org -
Posted in DebugWare Patterns, Debugging, Software Architecture, Software Engineering, Software Technical Support, Tools | 2 Comments »
Thursday, October 22nd, 2009
One ETW trace pointed to a set of intermittent symptoms (messages were simplified for this post):
# PID TID Message
[...]
31278 2300 7060 RequestXMLData entry
31281 2300 7060 RequestXMLData: XML error
[...]
Searching for issues having this error only pointed to a case with a mixed software product environment where some servers had the product version X and other servers the product version X+1. However, in the new case the customer claimed that he had only the product version X+1 on all production servers. We insisted and, after the closer inspection, servers with the product X were found…
- Dmitry Vostokov @ TraceAnalysis.org -
Posted in Debugging, Science of Software Tracing, Software Technical Support, Software Trace Analysis, Troubleshooting Methodology | No Comments »