Archive for the ‘Troubleshooting Methodology’ Category
Saturday, January 1st, 2011
With the new year starts the new initiative to integrate traditional multidisciplinary debugging approaches and methodologies with multiplatform pattern-driven software problem solving, unified debugging patterns, best practices in memory dump analysis and software tracing, computer security, economics, and the new emerging trends I’m going to write about during this year.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Best Practices, Computer Forensics, Computer Science, Countefactual Debugging, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Industry, Debugging Methodology, Debugging Trends, Dublin School of Security, Economics, Education and Research, Escalation Engineering, First Fault Problem Solving, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Memory Analysis Report System, Memory Dump Analysis Services, Memory Systems Language, Memory Visualization, New Debugging School, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Behavior Patterns, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Trace Analysis, Software Trace Visualization, Software Tracing Implementation Patterns, Software Troubleshooting Patterns, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tool Objects, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Unified Debugging Patterns, Victimware, Visual Dump Analysis, Webinars, Workaround Patterns | No Comments »
Wednesday, December 29th, 2010
In these post series we are going to discuss the best practices for software tracing implementation including appropriate patterns and their links to software trace analysis patterns. The first one is called Period Timestamp where the start and the end time (and the date if necessary) are recorded in the trace file. This helps in Inter-Correlation and News Value analysis between several different trace types. For example, in one scenario, we had WindowHistory and MessageHistory logs. We identified a problem in the former log as happening at this time:
Handle: 00010196 Class: "ClassA" Title: "TitleA"
Captured at: 13:36:30:533
[…]
However, when we looked at the latter trace to search for specific window messages posted or sent before that time we saw that the recording started later than the former event:
Start time: 13:36:35:830
Period timestamps are necessary to distinguish Incomplete History from Truncated Trace where in the former case the absence of expected trace message is due to some problem.
From a unified debugging patterns perspective we have this sequence fragment:
Implementation Patterns: Period Timestamp
Usage Patterns: Trace Simultaneously
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Best Practices, Debugging, Debugging Methodology, Software Architecture, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Reading, Software Tracing Implementation Patterns, Trace Analysis Patterns, Troubleshooting Methodology, Unified Debugging Patterns | No Comments »
Friday, December 17th, 2010
As the new decade is approaching (2011-2020) we would like to make a few previews and predictions:
- Increased complexity of software will bring more methods from biological, social sciences and humanities in addition to existing methods of automated debugging and computer science techniques
- Focus on first fault software problem solving (when aspect)
- Focus on pattern-driven software problem solving (how aspect)
- Fusion of debugging and malware analysis into a unified structural and behavioral pattern framework
- Visual debugging, memory and software trace visualization techniques
- Software maintenance certification
- Focus on domain-driven troubleshooting and debugging tools as a service (debugware TaaS)
- Focus on security issues related to memory dumps and software traces
- New scripting languages and programming language extensions for debugging
- The maturation of the science of memory snapshots and software traces (memoretics)
Imagining is not not limited to the above and more to come and explain in the forthcoming parts.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Anthropology, Archaeology of Computer Memory, Biology, Certification, Computer Science, Countefactual Debugging, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Industry, Debugging Methodology, Debugging Trends, Dublin School of Security, EasyDbg, Economics, Escalation Engineering, Evolution, First Fault Problem Solving, Futuristic Memory Dump Analysis, General Memory Analysis, General Science, Geography, Hardware, Hermeneutics of Memory Dumps and Traces, History, Humanities, Language, Laws of Troubleshooting and Debugging, Logic, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Analysis Forensics and Intelligence, Memory Analysis Report System, Memory Dump Analysis and History, Memory Systems Language, Memory Visualization, Paleo-debugging, Pattern Models, Pattern Prediction, Physics, Psychology, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Social Media, Social Sciences, Software Architecture, Software Behavior Patterns, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narrative Fiction, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Linguistics, Software Trace Visualization, Software Troubleshooting Patterns, Software Victimology, Statistics, Structural Memory Analysis and Social Sciences, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, Unified Debugging Patterns, Uses of UML, Victimware, Virtualization, Visual Dump Analysis, Windows System Administration, Workaround Patterns | No Comments »
Friday, November 12th, 2010
Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca
The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:
- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4
Product information:
- Title: Memory Dump Analysis Anthology, Volume 5
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 400 pages
- Publisher: Opentask (10 December 2010)
- ISBN-13: 978-1-906717-96-4
- Hardcover: 400 pages
- Publisher: Opentask (10 December 2010)
- ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Aesthetics of Memory Dumps, Announcements, Archaeology of Computer Memory, Art, Assembly Language, Books, C and C++, CDF Analysis Tips and Tricks, Categorical Debugging, Common Mistakes, Complete Memory Dump Analysis, Computer Science, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump De-analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Debugging Slang, Deep Down C++, Dr. Watson, Dublin School of Security, Education and Research, Escalation Engineering, Fun with Crash Dumps, Fun with Debugging, Fun with Software Traces, General Memory Analysis, Hermeneutics of Memory Dumps and Traces, Images of Computer Memory, Kernel Development, Malware Analysis, Malware Patterns, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Analysis Report System, Memory Diagrams, Memory Dreams, Memory Dump Analysis Jobs, Memory Dump Analysis Services, Memory Dump Analysis and History, Memory Dumps in Movies, Memory Dumps in Myths, Memory Religion (Memorianity), Memory Space Art, Memory Systems Language, Memory Visualization, Memory and Glitches, Memuonics, Metaphysical Society of Ireland, Minidump Analysis, Movies and Debugging, Multithreading, Museum of Debugging, Music for Debugging, Music of Computation, New Acronyms, New Words, Paleo-debugging, Pattern Models, Pattern Prediction, Philosophy, Physicalist Art, Psychoanalysis of Software Maintenance and Support, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Behavior Patterns, Software Chorography, Software Chorology, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Analysis and History, Software Trace Deconstruction, Software Trace Reading, Software Trace Visualization, Software Tracing for Dummies, Software Troubleshooting Patterns, Software Victimology, Stack Trace Collection, Structural Memory Analysis and Social Sciences, Structural Memory Patterns, Structural Trace Patterns, Systems Thinking, Testing, Theology, Tool Objects, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Uses of UML, Victimware, Virtualization, Vista, Visual Dump Analysis, Webinars, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns, x64 Windows | No Comments »
Saturday, November 6th, 2010
I’m pleased to announce that MDAA, Volume 4 is available in PDF format:
www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4
It features:
- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art
Its table of contents is available here:
http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf
Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Aesthetics of Memory Dumps, Announcements, AntiPatterns, Art, Assembly Language, Books, C and C++, CDF Analysis Tips and Tricks, Categorical Debugging, Common Mistakes, Complete Memory Dump Analysis, Computer Science, Countefactual Debugging, Crash Dump Analysis, Crash Dump Patterns, DebugWare Patterns, Debugging, Debugging Slang, Deep Down C++, Education and Research, Escalation Engineering, Fun with Crash Dumps, Fun with Debugging, Images of Computer Memory, Kernel Development, Memiotics (Memory Semiotics), Memoidealism, Memoretics, Memory Space Art, Memory Visualization, Memuonics, Metaphysics of Memory Worldview, Multithreading, Opcodism, Philosophy, Physicalist Art, Publishing, Science Fiction, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Behavior Patterns, Software Defect Construction, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Reading, Software Victimology, Stack Trace Collection, Testing, Tools, Trace Analysis Patterns, Troubleshooting Methodology, Uses of UML, Victimware, Virtualization, Vista, Visual Dump Analysis, WinDbg Scripts, WinDbg Tips and Tricks, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns, x64 Windows | No Comments »
Tuesday, October 26th, 2010
Sometimes when looking at a module list (lmv WinDbg command) we see the presence of the whole Pervasive System. It is not just a module that does function and/or message hooking but the whole system of modules from a single vendor that is context-aware (for example, reads its configuration from registry) and consists of several components that communicate with other processes. The penetrating system is supposed to add some additional value or to coexist peacefully in a larger environment. The system thus becomes coupled strongly and/or weakly with other processes it was never intended to work with as opposed to intended module variety. At one extreme modules from pervasive system can be ubiquitous and at the other end hidden. In such cases troubleshooting consists of the total removal of pervasive modules and if the problem disappears their exclusion one by one to find the problem component.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Troubleshooting Methodology | 1 Comment »
Friday, October 8th, 2010
I aim to release a beta version of that unified troubleshooting system together with EasyDbg (as WinDbg add-on) in a few months after finishing porting it to x64 platform. The system is very customizable and similar buttons can be created to launch troubleshooting tools and execute troubleshooting units of work for any client and server applications. It will also be covered in the forthcoming Software Troubleshooting and Debugging Tools presentation.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Debugging, Debugging Methodology, EasyDbg, Escalation Engineering, Memory Dump Analysis Services, Software Technical Support, Tool Objects, Tools, Troubleshooting Methodology | No Comments »
Tuesday, October 5th, 2010
The choice of webinars below mostly reflects my personal preferences and long time desire to speak on topics like systems thinking, troubleshooting tool design and development. The other topic about BSOD minidump analysis was requested by participants in an ongoing survey. There will be more topics in 2011. All forthcoming webinars will be hosted by Memory Dump Analysis Services. The planning list includes:
Systems Thinking in Memory Dump and Software Trace Analysis
Software Troubleshooting and Debugging Tools: Objects, Components, Patterns and Frameworks with UML
- UML basics
- DebugWare patterns
- Unified Troubleshooting Framework
- RADII software development process
- Hands-on exercise: designing and building a tool
Blue Screen Of Death Analysis Done Right: Minidump Investigation for System Administrators
- Making sense of !analyze –v output
- Get extra troubleshooting information with additional WinDbg commands
- Guessing culprits with raw stack analysis
- Who’s responsible: hardware or software?
- Checklist and patterns
- Including hands-on exercises: send your own minidumps
More detailed information will be available soon.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, CDF Analysis Tips and Tricks, Common Mistakes, Common Questions, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump De-analysis, Crash Dump Patterns, Crash Dumps for Dummies, DebugWare Patterns, Debugging, Debugging Industry, Debugging Methodology, Escalation Engineering, General Memory Analysis, Hermeneutics of Memory Dumps and Traces, Memory Systems Language, Minidump Analysis, Software Architecture, Software Behavior Patterns, Software Engineering, Software Technical Support, Software Trace Analysis, Software Tracing for Dummies, Structural Memory Patterns, Systems Thinking, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Vista, Webinars, WinDbg Tips and Tricks, Windows 7, Windows Server 2008, Windows System Administration, x64 Windows | No Comments »
Monday, September 27th, 2010
You are probably aware of Application and Driver Verifiers (including gflags.exe tool from Debugging Tools for Windows). These tools set flags that modify the behaviour of the system that is reflected in additional information being collected such as memory allocation history and in WinDbg output changes such as stack traces. These tools belong to a broad class of instrumentation tools and I call the analysis pattern Instrumentation Information. To check in a minidump, kernel and complete memory dumps whether Driver Verifier was enabled we use !verifier WinDbg command:
1: kd> !verifier
Verify Level 0 ... enabled options are:
Summary of All Verifier Statistics
RaiseIrqls 0x0
AcquireSpinLocks 0x0
Synch Executions 0x0
Trims 0x0
Pool Allocations Attempted 0x0
Pool Allocations Succeeded 0x0
Pool Allocations Succeeded SpecialPool 0x0
Pool Allocations With NO TAG 0x0
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0
Current paged pool allocations 0x0 for 00000000 bytes
Peak paged pool allocations 0x0 for 00000000 bytes
Current nonpaged pool allocations 0x0 for 00000000 bytes
Peak nonpaged pool allocations 0x0 for 00000000 bytes
0: kd> !verifier
Verify Level 3 ... enabled options are:
Special pool
Special irql
Summary of All Verifier Statistics
RaiseIrqls 0xdea5
AcquireSpinLocks 0x87b5c
Synch Executions 0x17b5
Trims 0xab36
Pool Allocations Attempted 0x8990e
Pool Allocations Succeeded 0x8990e
Pool Allocations Succeeded SpecialPool 0x29c0
Pool Allocations With NO TAG 0x1
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0
Current paged pool allocations 0x0 for 00000000 bytes
Peak paged pool allocations 0x0 for 00000000 bytes
Current nonpaged pool allocations 0x0 for 00000000 bytes
Peak nonpaged pool allocations 0x0 for 00000000 bytes
To check in a process user dump that Application Verifier (and gflags) was enabled use !avrf and !gflags WinDbg extension commands:
0:001> !avrf
Application verifier is not enabled for this process.
Page heap has been enabled separately.
0:001> !gflag
Current NtGlobalFlag contents: 0x02000000
hpa - Place heap allocations at ends of pages
Here is an example of an instrumented stack trace:
68546e88 verifier!AVrfpDphFindBusyMemoryNoCheck+0xb8
68546f95 verifier!AVrfpDphFindBusyMemory+0×15
68547240 verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0×20
68549080 verifier!AVrfDebugPageHeapFree+0×90
77190aac ntdll!RtlDebugFreeHeap+0×2f
7714a8ff ntdll!RtlpFreeHeap+0×5d
770f2a32 ntdll!RtlFreeHeap+0×142
75fb14d1 kernel32!HeapFree+0×14
748d4c39 msvcr80!free+0xcd
[…]
00a02bb2 ServiceA!ServiceMain+0×302
767175a8 sechost!ScSvcctrlThreadA+0×21
75fb3677 kernel32!BaseThreadInitThunk+0xe
770f9d42 ntdll!__RtlUserThreadStart+0×70
770f9d15 ntdll!_RtlUserThreadStart+0×1b
Another example that shows instrumentation difference. We run double free fault modeling application and see its stack trace from a crash dump:
0:000> !gflag
Current NtGlobalFlag contents: 0x00000000
0:000> kL 100
Child-SP RetAddr Call Site
00000000`002dec38 00000000`77735ce2 ntdll!NtWaitForSingleObject+0xa
00000000`002dec40 00000000`77735e85 ntdll!RtlReportExceptionEx+0x1d2
00000000`002ded30 00000000`77735eea ntdll!RtlReportException+0xb5
00000000`002dedb0 00000000`77736d25 ntdll!RtlpTerminateFailureFilter+0x1a
00000000`002dede0 00000000`77685148 ntdll!RtlReportCriticalFailure+0x96
00000000`002dee10 00000000`776a554d ntdll!_C_specific_handler+0x8c
00000000`002dee80 00000000`77685d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`002deeb0 00000000`776862ee ntdll!RtlDispatchException+0x3cb
00000000`002df590 00000000`77736cd2 ntdll!RtlRaiseException+0x221
00000000`002dfbd0 00000000`77737396 ntdll!RtlReportCriticalFailure+0x62
00000000`002dfca0 00000000`777386c2 ntdll!RtlpReportHeapFailure+0x26
00000000`002dfcd0 00000000`7773a0c4 ntdll!RtlpHeapHandleError+0x12
00000000`002dfd00 00000000`776dd1cd ntdll!RtlpLogHeapFailure+0xa4
00000000`002dfd30 00000000`77472c7a ntdll! ?? ::FNODOBFM::`string'+0x123b4
00000000`002dfdb0 00000000`6243c7bc kernel32!HeapFree+0xa
00000000`002dfde0 00000001`3f8f1033 msvcr90!free+0x1c
00000000`002dfe10 00000001`3f8f11f2 InstrumentedApp!wmain+0x33
00000000`002dfe50 00000000`7746f56d InstrumentedApp!__tmainCRTStartup+0x11a
00000000`002dfe80 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`002dfeb0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
Then we enable Application Verifier and full page heap in gflags.exe GUI. Actually 2 crash dumps are saved at the same time (we’d set up LocalDumps registry key on x64 W2K8 R2) with slightly different stack traces:
0:000> !gflag
Current NtGlobalFlag contents: 0x02000100
vrf - Enable application verifier
hpa - Place heap allocations at ends of pages
0:000> kL 100
Child-SP RetAddr Call Site
00000000`0022e438 00000000`77735ce2 ntdll!NtWaitForSingleObject+0xa
00000000`0022e440 00000000`77735e85 ntdll!RtlReportExceptionEx+0x1d2
00000000`0022e530 000007fe`f3ed26fb ntdll!RtlReportException+0xb5
00000000`0022e5b0 00000000`77688a8f verifier!AVrfpVectoredExceptionHandler+0×26b
00000000`0022e640 00000000`776859b2 ntdll!RtlpCallVectoredHandlers+0xa8
00000000`0022e6b0 00000000`776bfe48 ntdll!RtlDispatchException+0×22
00000000`0022ed90 000007fe`f3eca668 ntdll!KiUserExceptionDispatcher+0×2e
00000000`0022f350 000007fe`f3ec931d verifier!VerifierStopMessage+0×1f0
00000000`0022f400 000007fe`f3ec9736 verifier!AVrfpDphReportCorruptedBlock+0×155
00000000`0022f4c0 000007fe`f3ec99cd verifier!AVrfpDphCheckNormalHeapBlock+0xce
00000000`0022f530 000007fe`f3ec873a verifier!AVrfpDphNormalHeapFree+0×29
00000000`0022f560 00000000`7773c415 verifier!AVrfDebugPageHeapFree+0xb6
00000000`0022f5c0 00000000`776dd0fe ntdll!RtlDebugFreeHeap+0×35
00000000`0022f620 00000000`776c2075 ntdll! ?? ::FNODOBFM::`string’+0×122e2
00000000`0022f960 000007fe`f3edf4e1 ntdll!RtlFreeHeap+0×1a2
00000000`0022f9e0 00000000`77472c7a verifier!AVrfpRtlFreeHeap+0xa5
00000000`0022fa80 000007fe`f3ee09ae kernel32!HeapFree+0xa
00000000`0022fab0 00000000`642bc7bc verifier!AVrfpHeapFree+0xc6
00000000`0022fb40 00000001`3fac1033 msvcr90!free+0×1c
00000000`0022fb70 00000001`3fac11f2 InstrumentedApp!wmain+0×33
00000000`0022fbb0 00000000`7746f56d InstrumentedApp!__tmainCRTStartup+0×11a
00000000`0022fbe0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0022fc10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
0:000> kL 100
Child-SP RetAddr Call Site
00000000`0022e198 000007fe`f3ee0f82 ntdll!NtWaitForMultipleObjects+0xa
00000000`0022e1a0 000007fe`fd8513a6 verifier!AVrfpNtWaitForMultipleObjects+0×4e
00000000`0022e1e0 000007fe`f3ee0e2d KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`0022e2e0 000007fe`f3ee0edd verifier!AVrfpWaitForMultipleObjectsExCommon+0xad
00000000`0022e320 00000000`77473143 verifier!AVrfpKernelbaseWaitForMultipleObjectsEx+0×2d
00000000`0022e370 00000000`774e9025 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`0022e400 00000000`774e91a7 kernel32!WerpReportFaultInternal+0×215
00000000`0022e4a0 00000000`774e91ff kernel32!WerpReportFault+0×77
00000000`0022e4d0 00000000`774e941c kernel32!BasepReportFault+0×1f
00000000`0022e500 00000000`7770573c kernel32!UnhandledExceptionFilter+0×1fc
00000000`0022e5e0 00000000`77685148 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`0022e610 00000000`776a554d ntdll!_C_specific_handler+0×8c
00000000`0022e680 00000000`77685d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`0022e6b0 00000000`776bfe48 ntdll!RtlDispatchException+0×3cb
00000000`0022ed90 000007fe`f3eca668 ntdll!KiUserExceptionDispatcher+0×2e
00000000`0022f350 000007fe`f3ec931d verifier!VerifierStopMessage+0×1f0
00000000`0022f400 000007fe`f3ec9736 verifier!AVrfpDphReportCorruptedBlock+0×155
00000000`0022f4c0 000007fe`f3ec99cd verifier!AVrfpDphCheckNormalHeapBlock+0xce
00000000`0022f530 000007fe`f3ec873a verifier!AVrfpDphNormalHeapFree+0×29
00000000`0022f560 00000000`7773c415 verifier!AVrfDebugPageHeapFree+0xb6
00000000`0022f5c0 00000000`776dd0fe ntdll!RtlDebugFreeHeap+0×35
00000000`0022f620 00000000`776c2075 ntdll! ?? ::FNODOBFM::`string’+0×122e2
00000000`0022f960 000007fe`f3edf4e1 ntdll!RtlFreeHeap+0×1a2
00000000`0022f9e0 00000000`77472c7a verifier!AVrfpRtlFreeHeap+0xa5
00000000`0022fa80 000007fe`f3ee09ae kernel32!HeapFree+0xa
00000000`0022fab0 00000000`642bc7bc verifier!AVrfpHeapFree+0xc6
00000000`0022fb40 00000001`3fac1033 msvcr90!free+0×1c
00000000`0022fb70 00000001`3fac11f2 InstrumentedApp!wmain+0×33
00000000`0022fbb0 00000000`7746f56d InstrumentedApp!__tmainCRTStartup+0×11a
00000000`0022fbe0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0022fc10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
We also see above that enabling instrumentation triggers debug functions of runtime heap (RtlDebugFreeHeap).
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology, Troubleshooting Methodology | 1 Comment »
Sunday, July 18th, 2010

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free webinar
Date: 18th of August 2010
Time: 21:00 (BST) 16:00 (Eastern) 13:00 (Pacific)
Duration: 90 minutes
Topics include:
- User vs. kernel vs. physical (complete) memory space
- Challenges of complete memory dump analysis
- Common WinDbg commands
- Patterns
- Common mistakes
- Fiber bundles
- Hands-on exercise: a complete memory dump analysis
- A guide to DumpAnalysis.org case studies
Prerequisites: working knowledge of basic user process and kernel memory dump analysis or live debugging using WinDbg
The webinar link will be posted before 18th of August on DumpAnalysis.com
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Common Mistakes, Common Questions, Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Escalation Engineering, Memory Dump Analysis Services, Pattern Models, Security, Software Architecture, Software Behavior Patterns, Software Defect Construction, Software Engineering, Software Technical Support, Stack Trace Collection, Testing, Tools, Training and Seminars, Troubleshooting Methodology, Virtualization, Vista, Webinars, WinDbg Scripts, WinDbg Tips and Tricks, Windows 7, Windows Server 2008, Windows System Administration, x64 Windows | 1 Comment »
Sunday, May 2nd, 2010
c’t – Magazin für Computertechnik has published a review of First Fault Software Problem Solving book:
http://www.heise.de/ct/inhalt/2010/08/192/ (in German)
Fabian Röken kindly translated it into English:
No single large software package comes without errors. It seems that customers simply accept this, patiently waiting and hoping for patches or updates. Skwire sticks up for a more target-aimed approach: one will never get a faultless software, but it would already be a great improvement if flaws were already solved on their first occurrence (”first fault”) and not only after a long analysis (”second fault”).
The advantages are actually obvious. However, a corresponding stringent system architecture, as common on mainframes such as IBM’s z/OS, did not become prevalent in the PC market.
Skwire outlines the types of errors and strategies to resolve them in all details. His 40 years of experience, such as at IBM, shimmers through again and again. He puts emphasis on making sure that the reader understands the terminology he is using: “What is a problem in the first place?”, “What is a service point?” - in some cases he also explains specific metrics such as the “serviceability rating”.
His tool classification includes teaching tips, e.g. regarding the structure of a protocol in case of errors; or for tracking the important information how often an error must occur before a solution has to be approached. His suggestions equally address developers, designers, testers, managers - and the end user. In his last chapter he presents and reviews commercial tools in the first fault and second fault environment.
Skwire addresses a topic which is unfortunately very much neglected, and this alone already makes it worth enough to take a look at his book (***). Short quotations and humorous drawings relax the technical topic. If you are looking for an overview then you will be fine with this book. However, if you are a software developer looking for source code samples then you will search in vain. Skwire has released the book under the print-on-demand process. You will find it on Amazon, for example.
(Tobias Engler/fm)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Books, Escalation Engineering, First Fault Problem Solving, Software Engineering, Software Technical Support, Troubleshooting Methodology, Windows System Administration | No Comments »
Tuesday, April 20th, 2010
The following tool published by Citrix follows DebugWare patterns in its overall architecture and design and was implemented by a team of engineers using RADII process:
SsOnExpert - Single Sign-On XenApp Plug-in Troubleshooting Tool
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Citrix, DebugWare Patterns, Debugging, Debugging Methodology, Escalation Engineering, Software Architecture, Software Engineering, Software Maintenance Institute, Software Technical Support, Tools, Troubleshooting Methodology | No Comments »
Sunday, April 18th, 2010
OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.
Product information:
- Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 1600 pages
- Publisher: Opentask (31 May 2010)
- ISBN-13: 978-1-906717-99-5
Information about individual volumes:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Aesthetics of Memory Dumps, Announcements, AntiPatterns, Art, Assembly Language, Books, Bugchecks Depicted, C and C++, CDA Pattern Classification, CDF Analysis Tips and Tricks, Categorical Debugging, Certification, Citrix, Code Reading, Common Mistakes, Computational Ghosts and Bug Hauntings, Computer Science, Countefactual Debugging, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, Data Recovery, DebugWare Patterns, Debugging, Debugging Methodology, Debugging Slang, Deep Down C++, Dictionary of Debugging, Dr. Watson, Education and Research, Escalation Engineering, Fun with Crash Dumps, Futuristic Memory Dump Analysis, GDB for WinDbg Users, Hardware, History, Horrors of Computation, Hyper-V, Intelligent Memory Movement, Kernel Development, Laws of Troubleshooting and Debugging, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Auralization, Memory Dump Analysis Jobs, Memory Religion (Memorianity), Memory Space Art, Memory Space Music, Memory Systems Language, Memory Visualization, Memuonics, Minidump Analysis, Multithreading, Music for Debugging, Music of Computation, New Acronyms, New Words, Occult Debugging, Philosophy, Psi-computation, Publishing, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Visualization, Software Tracing for Dummies, Software Victimology, Stack Trace Collection, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Victimware, Virtualization, Vista, Visual Dump Analysis, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration | No Comments »
Friday, April 9th, 2010
Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):
- Are you interested in on-site training, prefer traveling or attending webinars?
- Are you interested in software trace analysis as well?
- What specific topics are you interested in?
- What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)
Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.
Looking forward to your responses. Any suggestions are welcome.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Announcements, AntiPatterns, Assembly Language, Bugchecks Depicted, C and C++, CDA Pattern Classification, CDF Analysis Tips and Tricks, Certification, Code Reading, Common Mistakes, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, Data Recovery, DebugWare Patterns, Debugging, Debugging Methodology, Deep Down C++, Education and Research, Escalation Engineering, FreeBSD Crash Corner, GDB for WinDbg Users, Hardware, Hyper-V, Java Debugging, Kernel Development, Laws of Troubleshooting and Debugging, Linux Crash Corner, Mac Crash Corner, Memory Analysis Forensics and Intelligence, Memory Visualization, Minidump Analysis, Multithreading, Security, Software Architecture, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Visualization, Software Tracing for Dummies, Software Victimology, Stack Trace Collection, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Victimware, Virtualization, Vista, Visual Dump Analysis, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns | No Comments »
Monday, April 5th, 2010
I’m very pleased to announce that the Korean edition is available:

The book can be found on:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, AntiPatterns, Assembly Language, Books, Books (Korean), C and C++, Computer Science, Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, Debugging, Dr. Watson, Escalation Engineering, Fun with Crash Dumps, GDB for WinDbg Users, Hardware, Kernel Development, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoretics, Memory Space Music, Memory Visualization, Minidump Analysis, Multithreading, Publishing, Science of Memory Dump Analysis, Security, Software Architecture, Software Engineering, Software Technical Support, Testing, Tools, Troubleshooting Methodology, Virtualization, Vista, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows Server 2008, Windows System Administration | No Comments »
Saturday, March 13th, 2010
This is a new methodology I’m working upon. The idea came from reading “About the Author” page in a book I got yesterday in my post:
The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting


I post a review here and on Amazon when finished reading. Just a few words now. This is the first career book I’m reading where I find pages in roman numerals useful. The page xiii itself looks like a good template (or an example) for a business-oriented CV summary. Thinking now about updating my CV book (2nd edition?):
Resume and CV: As a Book

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Books, Business, Crash Dump Analysis, Debugging, Debugging Methodology, Troubleshooting Methodology | 1 Comment »
Thursday, February 11th, 2010
This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:
- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes
Product information:
- Title: Memory Dump Analysis Anthology, Volume 4
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 410 pages
- Publisher: Opentask (30 March 2010)
- ISBN-13: 978-1-906717-86-5
- Hardcover: 410 pages
- Publisher: Opentask (30 April 2010)
- ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Aesthetics of Memory Dumps, Announcements, AntiPatterns, Art, Assembly Language, Books, C and C++, CDA Pattern Classification, CDF Analysis Tips and Tricks, Categorical Debugging, Certification, Citrix, Code Reading, Common Mistakes, Computer Science, Countefactual Debugging, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, Crash Dumps for Dummies, DebugWare Patterns, Debugging, Debugging Slang, Education and Research, Escalation Engineering, Fun with Crash Dumps, Games for Debugging, Hardware, Horrors of Computation, Hyper-V, Kernel Development, Laws of Troubleshooting and Debugging, Mathematics of Debugging, Memiotics (Memory Semiotics), Memoidealism, Memoretics, Memory Analysis Culture, Memory Analysis Forensics and Intelligence, Memory Dreams, Memory Dump Analysis Jobs, Memory Dumps in Movies, Memory Space Art, Memory Visualization, Memuonics, Minidump Analysis, Multithreading, Music for Debugging, New Acronyms, New Words, Opcodism, Philosophy, Physicalist Art, Publishing, Science Fiction, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Architecture, Software Defect Construction, Software Engineering, Software Generalist, Software Maintenance Institute, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Visualization, Software Tracing for Dummies, Software Victimology, Stack Trace Collection, Testing, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, Victimware, Virtualization, Vista, Visual Dump Analysis, WinDbg Scripts, WinDbg Tips and Tricks, WinDbg for GDB Users, Windows 7, Windows Server 2008, Windows System Administration, Workaround Patterns | No Comments »
Tuesday, January 26th, 2010
What happens when Hidden Output and Frozen Process patterns don’t help with annoying popup windows? The former can’t prevent windows from reappearing afresh and the latter could block other coupled processes that might exchange window messages with our suspended process or simply use any IPC mechanism. Here Axed Code pattern can help as demonstrated below. One process was frequently and briefly showing network disconnection message box or dialog. The problem is that it was also bringing its main window into foreground disrupting work in other windows because they were loosing focus. Next time the dialog appeared we found its process ID in Task Manager and attached WinDbg to it. We wasn’t sure what dialog function to intercept so we put a general breakpoint on all “Dialog” functions for all threads:
0:000:x86> bm *Dialog*
[...]
6: 73a8ba81 @!"MFC80!CDialog::~CDialog"
7: 73ac25e2 @!"MFC80!CPageSetupDialog::~CPageSetupDialog"
8: 73a94b6b @!"MFC80!CDHtmlDialog::_AfxSimpleScanf"
9: 73a8fbe9 @!"MFC80!CFileDialog::OnTypeChange"
10: 73a90b17 @!"MFC80!CColorDialog::GetRuntimeClass"
11: 73a8bb4a @!"MFC80!CDialog::CreateIndirect"
[...]
360: 73a93750 @!"MFC80!CDHtmlDialog::OnNavigateComplete"
361: 73a8f1f3 @!"MFC80!CCommonDialog::OnOK"
362: 73a95d9f @!"MFC80!CDHtmlDialog::GetDropTarget"
363: 73a90266 @!"MFC80!CPrintDialog::GetDevMode"
364: 73ac1514 @!"MFC80!COleInsertDialog::COleInsertDialog"
365: 73ac27c7 @!"MFC80!COlePropertiesDialog::COlePropertiesDialog"
366: 73a75282 @!"MFC80!CWnd::UpdateDialogControls"
367: 73a7fd86 @!"MFC80!CDialogBar::SetOccDialogInfo"
0:000:x86> g
Breakpoint 314 hit
MFC80!_AfxPostInitDialog:
73a7134e 55 push ebp
0:000:x86> kL 100
ChildEBP RetAddr Args to Child
0027ed2c 73a7180a MFC80!_AfxPostInitDialog
0027ed90 75628817 MFC80!_AfxActivationWndProc+0x90
0027edbc 7562898e USER32!InternalCallWinProc+0x23
0027ee34 7562c306 USER32!UserCallWinProcCheckWow+0x109
0027ee78 756375a2 USER32!SendMessageWorker+0x55b
0027ef4c 7563787a USER32!InternalCreateDialog+0xb64
0027ef70 75649b65 USER32!CreateDialogIndirectParamAorW+0x33
0027ef9c 75225192 USER32!CreateDialogParamA+0x4a
WARNING: Stack unwind information not available. Following frames may be wrong.
0027efc8 010c3bf1 DllA!WarningPopup+0×152
0027effc 73a71812 ProcessA+0×9fa1
00000000 00000000 MFC80!_AfxActivationWndProc+0×98
Now we cleared all breakpoints and put the new breakpoint on WarningPopup function:
0:000:x86> bc *
0:000:x86> bp DllA!WarningPopup
0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 51 push ecx
Then we assumed that the calling convention was the default one used by C or C++ code like _cdecl and took the bold step to replace push ecx with ret instruction:
0:000:x86> a 75225040
75225040 ret
ret
75225041
0:000:x86> g
Breakpoint 0 hit
DllA!WarningPopup:
75225040 c3 ret
0:000:x86> bc *
0:000:x86> g
Result: no warning popups anymore.
I originally intended to name the pattern Patched Code but then realized that code axing can also be done at the source code level as a quick temporal fix.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Debugging, Software Technical Support, Troubleshooting Methodology, WinDbg Tips and Tricks, Workaround Patterns | No Comments »
Monday, January 25th, 2010
Another workaround pattern for some problems is to freeze a process responsible for an annoying or excessive activity like in the case study: Debugger as a Shut Up Application. We can also use other tools for this purpose like Mark Russinovich’s PsSuspend. The suitable name for this pattern is Frozen Process.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Debugging, Software Technical Support, Tools, Troubleshooting Methodology, Workaround Patterns | 1 Comment »
Sunday, January 24th, 2010
After fighting HTML comments in Safari and Chrome (see the case study below) I came to an idea to name and catalog workaround patterns in troubleshooting and debugging. The first one is called Hidden Output. Sometimes we can just remove message boxes reporting minor problems and generating unnecessary support calls by hiding their windows, for example, by using CtxHideEx32. A different example is what I did today when troubleshooting Amazon aStore widget HTML code. It worked well in IE8:

However, in Apple Safari and Google Chrome the widget code was visible at the top of the page:
After a few unsuccessful attempts to debug the problem and faced with other pressing tasks I got a flash in my mind to hide the visible code by changing its color to be the same as its background:
<font color=”D3E7F4″><script type=”text/javascript”><!–
amazon_ad_tag=”crasdumpanala-20″;
amazon_ad_width=”728″;
amazon_ad_height=”90″;
amazon_color_background=”D3E7F4″;
amazon_color_border=”0000FF”;
amazon_color_logo=”FFFFFF”;
amazon_color_link=”0000FF”;
amazon_ad_logo=”hide”;
amazon_ad_link_target=”new”;
amazon_ad_border=”hide”;
amazon_ad_title=”OpenTask Books, Magazines and Notebooks”; //–></script>
<script type=”text/javascript” src=”http://www.assoc-amazon.com/s/asw.js”></script></font>
After that the picture became nicer:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Citrix, Debugging, Software Technical Support, Tools, Troubleshooting Methodology, Workaround Patterns | 1 Comment »