On The Same Page - coming to the same conclusion as another engineer when looking at a memory dump or a software trace. Literally means the same page of memory where an exception occurred or a stack trace is reconstructed or the same “page” when browsing a software trace output using a viewer.
Examples: Aha, we are on the same page!
With fix-privet,
Dr. DebugLove
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:
- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes
Product information:
Back cover features memory space art image: Internal Process Combustion.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
STUPID - Software Tracing Under Pressure Is Dumb
Examples: STUPID! STUPID! I told you to enable all modules! You included all but the one I need…
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
What happens when Hidden Output and Frozen Process patterns don’t help with annoying popup windows? The former can’t prevent windows from reappearing afresh and the latter could block other coupled processes that might exchange window messages with our suspended process or simply use any IPC mechanism. Here Axed Code pattern can help as demonstrated below. One process was frequently and briefly showing network disconnection message box or dialog. The problem is that it was also bringing its main window into foreground disrupting work in other windows because they were loosing focus. Next time the dialog appeared we found its process ID in Task Manager and attached WinDbg to it. We wasn’t sure what dialog function to intercept so we put a general breakpoint on all “Dialog” functions for all threads:
0:000:x86> bm *Dialog*
[...]
6: 73a8ba81 @!"MFC80!CDialog::~CDialog"
7: 73ac25e2 @!"MFC80!CPageSetupDialog::~CPageSetupDialog"
8: 73a94b6b @!"MFC80!CDHtmlDialog::_AfxSimpleScanf"
9: 73a8fbe9 @!"MFC80!CFileDialog::OnTypeChange"
10: 73a90b17 @!"MFC80!CColorDialog::GetRuntimeClass"
11: 73a8bb4a @!"MFC80!CDialog::CreateIndirect"
[...]
360: 73a93750 @!"MFC80!CDHtmlDialog::OnNavigateComplete"
361: 73a8f1f3 @!"MFC80!CCommonDialog::OnOK"
362: 73a95d9f @!"MFC80!CDHtmlDialog::GetDropTarget"
363: 73a90266 @!"MFC80!CPrintDialog::GetDevMode"
364: 73ac1514 @!"MFC80!COleInsertDialog::COleInsertDialog"
365: 73ac27c7 @!"MFC80!COlePropertiesDialog::COlePropertiesDialog"
366: 73a75282 @!"MFC80!CWnd::UpdateDialogControls"
367: 73a7fd86 @!"MFC80!CDialogBar::SetOccDialogInfo"
0:000:x86> g
Breakpoint 314 hit
MFC80!_AfxPostInitDialog:
73a7134e 55 push ebp
0:000:x86> kL 100
ChildEBP RetAddr Args to Child
0027ed2c 73a7180a MFC80!_AfxPostInitDialog
0027ed90 75628817 MFC80!_AfxActivationWndProc+0x90
0027edbc 7562898e USER32!InternalCallWinProc+0x23
0027ee34 7562c306 USER32!UserCallWinProcCheckWow+0x109
0027ee78 756375a2 USER32!SendMessageWorker+0x55b
0027ef4c 7563787a USER32!InternalCreateDialog+0xb64
0027ef70 75649b65 USER32!CreateDialogIndirectParamAorW+0x33
0027ef9c 75225192 USER32!CreateDialogParamA+0x4a
WARNING: Stack unwind information not available. Following frames may be wrong.
0027efc8 010c3bf1 DllA!WarningPopup+0×152
0027effc 73a71812 ProcessA+0×9fa1
00000000 00000000 MFC80!_AfxActivationWndProc+0×98
Now we cleared all breakpoints and put the new breakpoint on WarningPopup function:
0:000:x86> bc *
0:000:x86> bp DllA!WarningPopup
0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 51 push ecx
Then we assumed that the calling convention was the default one used by C or C++ code like _cdecl and took the bold step to replace push ecx with ret instruction:
0:000:x86> a 75225040
75225040 ret
ret
75225041
0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 c3 ret
0:000:x86> bc *
0:000:x86> g
Result: no warning popups anymore.
I originally intended to name the pattern Patched Code but then realized that code axing can also be done at the source code level as a quick temporal fix.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Another workaround pattern for some problems is to freeze a process responsible for an annoying or excessive activity like in the case study: Debugger as a Shut Up Application. We can also use other tools for this purpose like Mark Russinovich’s PsSuspend. The suitable name for this pattern is Frozen Process.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
After fighting HTML comments in Safari and Chrome (see the case study below) I came to an idea to name and catalog workaround patterns in troubleshooting and debugging. The first one is called Hidden Output. Sometimes we can just remove message boxes reporting minor problems and generating unnecessary support calls by hiding their windows, for example, by using CtxHideEx32. A different example is what I did today when troubleshooting Amazon aStore widget HTML code. It worked well in IE8:
However, in Apple Safari and Google Chrome the widget code was visible at the top of the page:
After a few unsuccessful attempts to debug the problem and faced with other pressing tasks I got a flash in my mind to hide the visible code by changing its color to be the same as its background:
<font color=”D3E7F4″><script type=”text/javascript”><!–
amazon_ad_tag=”crasdumpanala-20″;
amazon_ad_width=”728″;
amazon_ad_height=”90″;
amazon_color_background=”D3E7F4″;
amazon_color_border=”0000FF”;
amazon_color_logo=”FFFFFF”;
amazon_color_link=”0000FF”;
amazon_ad_logo=”hide”;
amazon_ad_link_target=”new”;
amazon_ad_border=”hide”;
amazon_ad_title=”OpenTask Books, Magazines and Notebooks”; //–></script>
<script type=”text/javascript” src=”http://www.assoc-amazon.com/s/asw.js”></script></font>
After that the picture became nicer:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
After exciting results of the previous year of debugging it is time to announce modest plans for this year, 0×7DA:
Release the first beta version of EasyDbg
Release the first beta version of CARE (Crash Analysis Report Environment) for a pattern-driven debugger log analyzer with standards for structured audience-driven reports
Release the first beta version of STARE (Software Trace Analysis Report Environment) for a pattern-driven software trace analyzer with corresponding standards for structured audience-driven reports
Publish the following books on dump analysis that address different audiences (general users, system administrators, support and escalation engineers, testers, software engineers, security and software defect researchers):
- Windows Debugging Notebook
- Crash Dump Analysis for System Administrators and Support Engineers
- Memory Dump Analysis Anthology, Volume 4
- Memory Dump Analysis Anthology, Volume 5
- Memory Dump Analysis Anthology Color Supplement
- Principles of Memory Dump Analysis
- My Computer Crashes and Freezes: A Non-technical Guide to Software and Hardware Errors
- Linux, FreeBSD and Mac OS X Debugging: Practical Foundations
- Encyclopedia of Crash Dump Analysis Patterns
- WinDbg In Use: Debugging Exercises
Publish articles related to memory dump analysis in Debugged! magazine
Update WinDbg Poster and Cards
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The Year of Debugging, 0×7D9, was a remarkable year for DumpAnalysis.org. Here is the list of achievements to report:
- Software Trace Analysis as a new discipline with its own set of patterns
- Unification of Memory Dump Analysis with Software Trace Analysis (DA+TA)
- New computer memory dump-based art movements: Opcodism and Physicalist Art
- Discovery of 3D computer memory visualization techniques
- Establishing Software Maintenance Institute
- Broadening software fault injection as Software Defect Construction discipline
- Establishing a new profession of a Software Defect Researcher
- Starting ambitious Dictionary of Debugging
- Publishing Windows Debugging: Practical Foundations book
- Publishing the first x86-free Windows debugging book: x64 Windows Debugging: Practical Foundations
- Establishing the new debugging magazine: Debugged! MZ/PE
- Publishing Memory Dump Analysis Anthology, Volume 3
- Cooperation with OpenTask to promote First Fault Software Problem Solving book
- Establishing Debugging Expert(s) Magazine Online
- Creating the first development process for debugging and software troubleshooting tools: RADII
- Publishing the first pattern-driven memory dump analysis troubleshooting methodology as a foundation for software debugging
- Proposal for an International Memory Analysts and Debuggers Day
- Almost completed Windows Debugging Notebook to be published soon
- The founder of DumpAnalysis.org (Dr. DebugLove) becomes a member of Citrix Systems Tweetrix Support Team
Now DumpAnalysis.org focuses on The Year of Dump Analysis, 0×7DA, as a foundation for the forthcoming debugging decade and reveals future plans this weekend.
I’m sure that many other organizations and individuals have no less remarkable accomplishments to report for 2009. I promise to track down and write about some of them in the forthcoming book:
The Science of Dr. Watson: An Illustrated History of Debugging (ISBN: 978-1906717070)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Main topics of Debugged! MZ/PE magazine issues for June and September 2010:
Debugged! MZ/PE: Multiplatform Software Defects, June, 2010 (Paperback, ISBN: 978-1906717902)
Debugged! MZ/PE: Software Defect Visualization and Annotation, September, 2010 (Paperback, ISBN: 978-1906717919)
Here are draft front covers designed today:
- Dmitry Vostokov @ DumpAnalysis.org -
See the greeting card on the portal together with New Year’s Eve code analysis puzzle:
DumpAnalysis.org Wishes Happy New Year 7DA!
- Dmitry Vostokov @ DumpAnalysis.org -
“Memory dumps are facts.”
I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:
Memory Dump Analysis Anthology, Volume 3
In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.
The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4.
- Dmitry Vostokov @ DumpAnalysis.org -
I propose to celebrate it on 08.08 every year starting from The Year of Dump Analysis, 2010, 7DA at 8:00 (I prefer 8pm for MAD Day, moderation is important in debugging too).
What do you think? If you count things from 0 or favour user space there is an alternative date: 07.07
- Dmitry Vostokov @ DumpAnalysis.org -
Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday. The SMI web site temporary points to Crash Dump Analysis and Debugging Portal: www.sminstitute.com
Here is the current component structure of various institutions (depicted in UML):
Interface Tags:
IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering
- Dmitry Vostokov @ DumpAnalysis.org -
Finally, after the long delay, the issue is available in print on Amazon and through other sellers:
Debugged! MZ/PE: Software Tracing
- Dmitry Vostokov @ DumpAnalysis.org -
In its simplest form the first law of troubleshooting and debugging states that:
The more frequent a problem is, the simpler tool is needed to resolve and fix it.
- Dmitry Vostokov @ DumpAnalysis.org -
I’m very pleased to announce that Dan Skwire’s unique book has been published by OpenTask:
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
- Dmitry Vostokov @ DumpAnalysis.org -
The new article has been added to the free online version of Debugged! magazine (www.debuggingexpert.com or www.debuggingexperts.com):
Memory Dump and Trace Analysis: A Unified Pattern Approach
- Dmitry Vostokov @ DumpAnalysis.org -
Suppose we want to write a support tool to troubleshoot a frequent software problem. Would we start with GUI? Most tools are command line tools and we therefore choose Tool Façade pattern to decouple GUI from CLI:
By making our GUI optional we postpone its development until later implementation phase when we want to beautify or simplify the tool user interface. The troubleshooting of software problem we want to automate is quite complex, scattered across several knowledge base articles, with many troubleshooting steps and various checks. We therefore treat different steps as TUWs, Troubleshooting Units of Work (aggregating related ones if necessary) and choose Checklist pattern to coordinate them:
Initially we think that just one checklist is suffice but while studying the domain of troubleshooting we find that the problem usually spans client and server computers, where servers sometimes have client roles as well and different troubleshooting steps and checks are needed. In order to simplify the tool we apply Span Differentiator pattern and our architectural UML component diagram becomes more refined:
We also see that different TUW use similar operations like querying registry values or checking whether certain processes are running and factor them out into separate SMs, Supporting Modules:
Deployment methods are important in troubleshooting where it is not always desirable to change configuration on production servers and we plan for optional Self Extractor pattern:
In diagrams we liberally mixed component and static UML diagramming elements. We also need to design a uniform interface that each TUW module exposes to the main tool coordinator module but we leave it for the part 2.
- Dmitry Vostokov @ DumpAnalysis.org -
Support tools can be packaged for delivery as (not counting optional separate readme and EULA files):
The latter common packaging choice is a pattern itself that I call Self Extractor. Typical examples include sysinternals tools like Process Monitor. Here is a simple unorthodox UML diagram for a single platform tool combining elements from static and component diagrams to illustrate component embedding (not sharing):
SM is a supporting module, for example, a driver, to carry out some task or receive notifications.
Some tools even extract different components depending on the processor architecture, for example, x86 and x64. This is illustrated on a more elaborate UML diagram:
The typical example would be a backward compatible x86 Extractor that extracts an x64 driver (Extractee) for an x64 OS kernel that doesn’t load x86 drivers.
- Dmitry Vostokov @ DumpAnalysis.org -
One ETW trace pointed to a set of intermittent symptoms (messages were simplified for this post):
# PID TID Message
[...]
31278 2300 7060 RequestXMLData entry
31281 2300 7060 RequestXMLData: XML error
[...]
Searching for issues having this error only pointed to a case with a mixed software product environment where some servers had the product version X and other servers the product version X+1. However, in the new case the customer claimed that he had only the product version X+1 on all production servers. We insisted and, after the closer inspection, servers with the product X were found…
- Dmitry Vostokov @ TraceAnalysis.org -
