Software Diagnostics Library

The following WER dialog appeared after I copied an RTF text from an e-mail client to IE while editing another blog post:

Although the dialog points to html.iec it is good to show the basic techniques for component identification using WinDbg. I took a manual user dump of that unresponsive process while it was showing that dialog above using Task Manager:

The saved process memory dump shows that IE was virtualized 32-bit process running on x64 Windows:

STACK_TEXT: 
00000000`000dee18 00000000`75bbab46 : wow64cpu!TurboDispatchJumpAddressEnd+0xef
00000000`000deec0 00000000`75bba14c : wow64!Wow64SystemServiceEx+0×27a
00000000`000deef0 00000000`778f52d3 : wow64!Wow64LdrpInitialize+0×4b4
00000000`000df450 00000000`778f5363 : ntdll!CsrClientConnectToServer+0×493
00000000`000df700 00000000`778e85ce : ntdll!CsrClientConnectToServer+0×523
00000000`000df7b0 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

However, even after switching to x86 mode, !analyze -v doesn’t report any signs of exception violation, c0000005, seen from WER dialog above:

0:000> .load wow64exts; .effmach x86
Effective machine: x86 compatible (x86)

0:000:x86> !analyze -v
[...]

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

STACK_TEXT: 
0027e428 7679e91a ntdll_77a60000!NtWaitForMultipleObjects+0x15
0027e4c4 76a58f76 kernel32!WaitForMultipleObjectsEx+0x11d
0027e518 75ad6071 user32!RealMsgWaitForMultipleObjectsEx+0x14d
WARNING: Frame IP not in any known module. Following frames may be wrong.
0027e538 75ad61f0 ieui+0x6071
0027e560 75ad6196 ieui+0x61f0
0027e59c 7291ffc6 ieui+0x6196
0027e5a8 7290f579 ieframe!BrowserThreadProc+0x3f
0027e5cc 7290f4c7 ieframe!BrowserNewThreadProc+0x7b
0027f63c 728fd1ba ieframe!SHOpenFolderWindow+0x188
0027f86c 013133c3 ieframe!IEWinMain+0x2d9
0027fcb0 0131325a iexplore!wWinMain+0x27b
0027fd44 7680e4a5 iexplore!_initterm_e+0x1b1
0027fd50 77adcfed kernel32!BaseThreadInitThunk+0xe
0027fd90 77add1ff ntdll_77a60000!__RtlUserThreadStart+0x23
0027fda8 00000000 ntdll_77a60000!_RtlUserThreadStart+0x1b

We list all threads for any exception processing signs and we find one such thread indeed:

0:002:x86> ~*kc 100

[...]

   2  Id: 404.ba4 Suspend: 0 Teb: 7efaa000 Unfrozen

ntdll_77a60000!NtWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!UnhandledExceptionFilter
ntdll_77a60000!__RtlUserThreadStart
ntdll_77a60000!_EH4_CallFilterFunc
ntdll_77a60000!_except_handler4
ntdll_77a60000!ExecuteHandler2
ntdll_77a60000!ExecuteHandler
ntdll_77a60000!KiUserExceptionDispatcher
WARNING: Frame IP not in any known module. Following frames may be wrong.
0×0
html!LwMultDivRU
html!FMarkListCallback
html!JcCalcFromXaExtents
html!EmitNonBreakingSpace
html!FEmitHtmlFnOtag
html!ConvertRtfToForeign
html!FceRtfToForeign
html!RtfToForeign32
mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml
mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml
mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml
mshtmled!CHTMLEditor::ConvertRTFToHTML
mshtmled!CPasteCommand::PasteFromClipboard
mshtmled!CPasteCommand::PrivateExec
mshtmled!CCommand::Exec
mshtmled!CMshtmlEd::Exec
mshtml!CEditRouter::ExecEditCommand
mshtml!CDoc::ExecHelper
mshtml!CFrameSite::Exec
mshtml!CDoc::RouteCTElement
mshtml!CDoc::ExecHelper
mshtml!CDoc::Exec
mshtml!CDoc::OnCommand
mshtml!CDoc::OnWindowMessage
mshtml!CServer::WndProc
user32!InternalCallWinProc
user32!UserCallWinProcCheckWow
user32!SendMessageWorker
user32!SendMessageW
mshtml!CElement::PerformTA
mshtml!CDoc::PerformTA
mshtml!CDoc::PumpMessage
mshtml!CDoc::DoTranslateAccelerator
mshtml!CServer::TranslateAcceleratorW
mshtml!CDoc::TranslateAcceleratorW
ieframe!CProxyActiveObject::TranslateAcceleratorW
ieframe!CDocObjectView::TranslateAcceleratorW
ieframe!CCommonBrowser::v_MayTranslateAccelerator
ieframe!CShellBrowser2::_MayTranslateAccelerator
ieframe!CShellBrowser2::v_MayTranslateAccelerator
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll_77a60000!__RtlUserThreadStart
ntdll_77a60000!_RtlUserThreadStart

[...]

The same stack fragment with function parameters has a zero for its UnhandledExceptionFilter first parameter, perhaps because of optimized code reusing local variable slots, and we cannot apply .exptr meta-command:

0:002:x86> ~2kv
ChildEBP RetAddr  Args to Child             
0361d938 7679e91a 00000002 0361d988 00000001 ntdll_77a60000!NtWaitForMultipleObjects+0x15
0361d9d4 767949d9 0361d988 0361da24 00000000 kernel32!WaitForMultipleObjectsEx+0x11d
0361d9f0 7684573d 00000002 0361da24 00000000 kernel32!WaitForMultipleObjects+0x18
0361da5c 76845969 0361db2c 00000001 00000001 kernel32!WerpReportFaultInternal+0x16d
0361da70 7681c66f 0361db2c 00000001 35137a6c kernel32!WerpReportFault+0x70
0361dafc 77add03e 00000000 77aaf2d0 00000000 kernel32!UnhandledExceptionFilter+0×1b5
0361db04 77aaf2d0 00000000 0361fe04 77a8da38 ntdll_77a60000!__RtlUserThreadStart+0×6f
0361db18 77b129b3 00000000 00000000 00000000 ntdll_77a60000!_EH4_CallFilterFunc+0×12
0361db40 77a83099 fffffffe 0361fdf4 0361dc7c ntdll_77a60000!_except_handler4+0×8e
0361db64 77a8306b 0361dc2c 0361fdf4 0361dc7c ntdll_77a60000!ExecuteHandler2+0×26
0361dc14 77a82eff 0161dc2c 0361dc7c 0361dc2c ntdll_77a60000!ExecuteHandler+0×24
0361dc18 0161dc2c 0361dc7c 0361dc2c 0361dc7c ntdll_77a60000!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0361df5c 72333b0e 07feaee4 0000001f 00000000 0×161dc2c
0361df74 72332ba9 00000000 0361dffc 00000000 html!LwMultDivRU+0×4b6
0361dfb8 7233518a 0361dffc 0000000c 0361e074 html!FMarkListCallback+0×56c
0361dfc8 723368b7 0000001f 0361dffc 0000000f html!JcCalcFromXaExtents+0×91
0361e074 723370c5 0000000f 00000000 000007d0 html!EmitNonBreakingSpace+0×445
0361e1a0 723407ff 00000000 0000000f 00000104 html!FEmitHtmlFnOtag+0×17d
0361e1c4 7231c6a8 00000001 00000000 00000000 html!ConvertRtfToForeign+0×105
0361e64c 7231c745 00ed03bc 00000000 00ed0374 html!FceRtfToForeign+0×266

Also the transition from html.iec module to exception processing code is through an invalid address 0×161dc2c so we might guess that this was an instance of wild code pointer or the case of incorrect stack trace. However using techniques to get exception context from hidden exceptions we get the following stack trace:

0:002:x86> .cxr 0361dc7c
eax=0000004e ebx=09d5755f ecx=07feaee4 edx=07feaf08 esi=07feaee4 edi=00000000
eip=7233d50b esp=0361df48 ebp=0361df5c iopl=0  nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
html!FPseudoStyleBis+0×26:
7233d50b 0fb70f          movzx   ecx,word ptr [edi]       ds:002b:00000000=????

0:002:x86> kL 100
ChildEBP RetAddr 
0361df50 7233d597 html!FPseudoStyleBis+0x26
0361df5c 72333b0e html!BisFromLpxszStyle+0x1c
0361df74 72332ba9 html!LwMultDivRU+0x4b6
0361dfb8 7233518a html!FMarkListCallback+0x56c
0361dfc8 723368b7 html!JcCalcFromXaExtents+0x91
0361e074 723370c5 html!EmitNonBreakingSpace+0x445
0361e1a0 723407ff html!FEmitHtmlFnOtag+0x17d
0361e1c4 7231c6a8 html!ConvertRtfToForeign+0x105
0361e64c 7231c745 html!FceRtfToForeign+0x266
0361e674 744e5ad4 html!RtfToForeign32+0x51
0361eabc 744e5c83 mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml+0x163
0361ef10 7449cc15 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x11a
0361ef2c 7449cd81 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x38
0361ef40 744cdcea mshtmled!CHTMLEditor::ConvertRTFToHTML+0x12
0361efac 744ce392 mshtmled!CPasteCommand::PasteFromClipboard+0x2c0
0361f01c 7448d218 mshtmled!CPasteCommand::PrivateExec+0x47a
0361f040 7448d1ad mshtmled!CCommand::Exec+0x4b
0361f064 74000d14 mshtmled!CMshtmlEd::Exec+0xf9
0361f094 73ff88a8 mshtml!CEditRouter::ExecEditCommand+0xd6
0361f43c 7415eccf mshtml!CDoc::ExecHelper+0x338d
0361f488 73ff8a2f mshtml!CFrameSite::Exec+0x264
0361f4bc 73ff87af mshtml!CDoc::RouteCTElement+0xf1
0361f854 73ff8586 mshtml!CDoc::ExecHelper+0x325e
0361f874 740a0e7b mshtml!CDoc::Exec+0x1e
0361f8ac 7401a708 mshtml!CDoc::OnCommand+0x9c
0361f9c0 73f297e1 mshtml!CDoc::OnWindowMessage+0x841
0361f9ec 76a58807 mshtml!CServer::WndProc+0x78
0361fa18 76a58962 user32!InternalCallWinProc+0x23
0361fa90 76a5c4b6 user32!UserCallWinProcCheckWow+0x109
0361fad4 76a5c517 user32!SendMessageWorker+0x55b
0361faf8 7400fb9b user32!SendMessageW+0x7f
0361fb24 73f68e5a mshtml!CElement::PerformTA+0x71
0361fb44 73f68db9 mshtml!CDoc::PerformTA+0xd8
0361fbc0 73ff381c mshtml!CDoc::PumpMessage+0x8e0
0361fc74 73ff3684 mshtml!CDoc::DoTranslateAccelerator+0x33f
0361fc90 73ff34cc mshtml!CServer::TranslateAcceleratorW+0x56
0361fcb0 7296f550 mshtml!CDoc::TranslateAcceleratorW+0x83
0361fccc 7296f600 ieframe!CProxyActiveObject::TranslateAcceleratorW+0x30
0361fcf0 7296fca1 ieframe!CDocObjectView::TranslateAcceleratorW+0xb1
0361fd10 7296faf4 ieframe!CCommonBrowser::v_MayTranslateAccelerator+0xda
0361fd3c 7296f7b0 ieframe!CShellBrowser2::_MayTranslateAccelerator+0x68
0361fd4c 7296f7f5 ieframe!CShellBrowser2::v_MayTranslateAccelerator+0x15
0361fdb8 7680e4a5 ieframe!CTabWindow::_TabWindowThreadProc+0x264
0361fdc4 77adcfed kernel32!BaseThreadInitThunk+0xe
0361fe04 77add1ff ntdll_77a60000!__RtlUserThreadStart+0x23
0361fe1c 00000000 ntdll_77a60000!_RtlUserThreadStart+0x1b

Then we can use lmv command to see the component version:

0:002:x86> lmv m html
start             end                 module name
722f0000 7236f000   html     # (pdb symbols)          c:\mss\html.pdb\34A9E6645AEA4E6A83EA51D2849C731C1\html.pdb
    Loaded symbol image file: html.iec
    Image path: C:\Windows\System32\html.iec
    Image name: html.iec
    Timestamp:        Tue Mar 03 03:26:33 2009 (49ACA369)
    CheckSum:         00086241
    ImageSize:        0007F000
    File version:     2017.0.0.18226
    Product version:  6.0.6001.18226
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     HTML.IEC
    OriginalFilename: HTML.IEC
    ProductVersion:   6.0.6001.18226
    FileVersion:      2017.0.0.18226 (vistasp1_gdr.090302-1506)
    FileDescription:  Microsoft HTML Converter
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org acquires TraceAnalysis.org to complete computer DATA artifact analysis. The domain currently points to Dump Analysis Portal page but this might change in the future.

- Dmitry Vostokov @ DumpAnalysis.org -

Next obvious structural trace analysis pattern is called Circular Trace. Sometimes this information is missing in the problem description or trace metadata doesn’t reflect this. Then circular traces can be detected by trace file size (usually large) and from timestamps, like this 100Mb CDF trace snippet:

No     Module  PID  TID  Date      Time         Statement
[Begin of trace listing]
1      ModuleA 4280 1736 5/28/2009 08:53:50.496 [... Trace statement 1]
2      ModuleB 6212 6216 5/28/2009 08:53:52.876 [... Trace statement 2]
3      ModuleA 4280 4776 5/28/2009 08:54:13.537 [... Trace statement 3]
[... Some traced exceptions helpful for analysis ...]
3799   ModuleA 4280 3776 5/28/2009 09:15:00.853 [… Trace statement 3799]
3800   ModuleA 4280 1736 5/27/2009 09:42:12.029 [… Trace statement 3800]
[… Skipped …]
[… Skipped …]
[… Skipped …]
579210 ModuleA 4280 4776 5/28/2009 08:53:35.989 [… Trace statement 579210]
[End of trace listing]

In such traces, the analysis region is usually found at the wrap point because, as soon as an elusive, hard-to-reproduce problem occurs, the trace is stopped.

- Dmitry Vostokov @ TraceAnalysis.org -

DATA (Dump Artifact + Trace Artifact) - > DATA (Dump Analysis + Trace Analysis) examples:

1.  Efficient

- My 64Gb server bluescreens. I set a complete memory dump option in Control Panel.

- A user cannot connect. I started tracing yesterday. Stopped today.

- I analyze all these artifacts every day.

2. Effective

- My 64Gb server bluescreens. I set a kernel memory dump option in Control Panel.

- A user cannot connect. I started tracing, tried to connect, stopped tracing.

- I analyze all these artifacts every day and write articles to reduce DATA load.

- Dmitry Vostokov @ DumpAnalysis.org -

Good software engineers write good software trace statements. Good software support engineers and responsible customers trace^(*) software wisely, enabling it at the right time and in the right quantities. The following preliminary article was written to help to trace software effectively to result in faster problem resolution via trace analysis:

Tracing Best Practices

Although, currently it’s geared towards CDF tracing in Citrix terminal services environment, these recommendations can be generalized to other traces as well and the article will be extended over time.

(*) Note the following terminological difference here. “Tracing” is meant in “select” / “start” / “stop” sense and not how to write good software trace statements during code construction and maintenance phases.

- Dmitry Vostokov @ TraceAnalysis.org -

The number of blog visits (excluding portal main page and other my blogs) was about 15,000 - 16,000 by the end of the last year and then it dropped to 13,000. I explain this as the fact that 5% - 10% of engineers were no longer interested in crash dumps and debugging due to layoffs. This month I see the number of visits exceeds 14,000 and this surely makes me more optimistic about the prospect of economic recovery:

- Dmitry Vostokov @ DumpAnalysis.org -

I was delighted to know that my books in paperback editions will be available in minutes via Espresso Book Machine: Source. You can read more about this ATM book machine from its inventor:

http://www.ondemandbooks.com/home.htm

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Literate Scientist blog (last month focus was mostly on physics):

Ideas That Matter

Linear Algebra Demystified

The 10,000 Year Explosion

Homework for Grown-ups

Einstein’s Mistakes

Relativity Demystified

Quantum Mechanics Demystified

30,000 Years of Art

Quantum Field Theory Demystified

Quantum Field Theory I

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org has changed its book store to Amazon aStore to incorporate all published OpenTask books, magazines and notebooks:

Here is the direct link:

Dump Analysis Portal Store

The screenshot:

 - Dmitry Vostokov @ DumpAnalysis.org -

I’ve been a member of Amazon Associates program since June 2007, providing links to books on my various blogs. Visitors did almost 20,000 clicks and bought almost 1,000 books (although not always ones that I recommended). Here is the list of bought or pre-ordered titles sorted by popularity:

Memory Dump Analysis Anthology, Volume 1
Advanced Windows Debugging (The Addison-Wesley Microsoft Technology Series)
Windows Debugging: Practical Foundations
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Windows Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (PRO-Developer)
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The IDA Pro Book: The Unofficial Guide to the World’s Most Popular Disassembler
Windows via C/C++ (Pro - Developer)
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
Windows Sysinternals Administrator’s Reference (Inside Out)
Microsoft Windows Internals (4th Edition): Microsoft Windows Server 2003, Windows XP, and Windows 2000
Running Xen: A Hands-On Guide to the Art of Virtualization
The Definitive Guide to the Xen Hypervisor (Prentice Hall Open Source Software Development Series)
Baby Turing
Inside the Machine: An Illustrated Introduction to Microprocessors and Computer Architecture
Reversing: Secrets of Reverse Engineering
The Developer’s Guide to Debugging
The Old New Thing: Practical Development Throughout the Evolution of Windows
Debugging Microsoft .NET 2.0 Applications
Developing Drivers with the Windows  Driver Foundation (Pro Developer)
The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System
Apache Server 2.0: The Complete Reference
Becoming a Technical Leader: An Organic Problem-Solving Approach
C++ Primer Plus (5th Edition)
DLL List Landscape: The Art from Computer Memory Space
Debugging by Thinking: A Multidisciplinary Approach (HP Technologies)
Designing Storage Area Networks: A Practical Reference for Implementing Fibre Channel and IP SANs (2nd Edition)
Hacking: The Art of Exploitation, 2nd Edition
Inside the C++ Object Model
Linux Kernel Development (2nd Edition) (Novell Press)
MCITP Self-Paced Training Kit (Exams 70-640, 70-642, 70-643, 70-647): Windows Server 2008 Enterprise Administrator Core Requirements
Microsoft Visual C# 2008 Step by Step
Programming Interviews Exposed: Secrets to Landing Your Next Job (Programmer to Programmer)
Programming Language Pragmatics, Second Edition
Python Essential Reference (3rd Edition) (Developer’s Library)
Reverse Engineering Code with IDA Pro
Ring Bearer, The
Rootkits: Subverting the Windows Kernel (Addison-Wesley Software Security Series)
Strong Women, Strong Bones, Updated
The Back of the Napkin: Solving Problems and Selling Ideas with Pictures
Windows NT/2000 Native API Reference (Circle)
Writing Secure Code for Windows Vista (Pro - Step By Step Developer)
A Time to Die
Accelerated Learning for the 21st Century: The Six-Step Plan to Unlock Your Master-Mind
Administering Windows Vista Security: The Big Surprises
Advanced Programming in the UNIX(R) Environment (2nd Edition) (Addison-Wesley Professional Computing Series)
Advanced Topics in Types and Programming Languages
All the Mathematics You Missed: But Need to Know for Graduate School
An Introduction to Lambda Calculi for Computer Scientists
Asterisk: The Future of Telephony
BIOS Disassembly Ninjutsu Uncovered (Uncovered series)
Basic Abstract Algebra
Basic Category Theory for Computer Scientists (Foundations of Computing)
Bit by Bit: An Illustrated History of Computers
Breaking Through the BIOS Barrier: The Definitive BIOS Optimization Guide for PCs
C++ GUI Programming with Qt4 (2nd Edition) (Prentice Hall Open Source Software Development Series)
C++ Iostreams Handbook
C++ Template Metaprogramming: Concepts, Tools, and Techniques from Boost and Beyond (C++ In-Depth Series)
C: A Reference Manual (5th Edition)
CLR via C#, Second Edition (Pro Developer)
Code Complete: A Practical Handbook of Software Construction
Code Craft: The Practice of Writing Excellent Code
Code: The Hidden Language of Computer Hardware and Software
Compilers: Principles, Techniques, and Tools (2nd Edition)
Complete Digital Photography, Fourth Edition (Graphics Series)
Computer Repair with Diagnostic Flowcharts: Troubleshooting PC Hardware Problems from Boot Failure to Poor Performance, Revised Edition
Computer Science Made Simple: Learn how hardware and software work– and how to make them work for you! (Made Simple)
Concepts, Techniques, and Models of Computer Programming
Cross-Platform Development in C++: Building Mac OS X, Linux, and Windows Applications
Data Mining and Knowledge Discovery Handbook
Debugging
Debugging Applications: Microsoft (Dv-Mps Programming)
Debugging Windows Programs: Strategies, Tools, and Techniques for Visual C++ Programmers (The DevelopMentor Series)
Developing Drivers with the Windows  Driver Foundation
Domain-Specific Development with Visual Studio DSL Tools (Microsoft .NET Development Series)
Electronic Data Interchange in Finance and Accounting
Exceptional C++: 47 Engineering Puzzles, Programming Problems, and Solutions (C++ In-Depth Series)
Expert F# (Expert’s Voice in .Net)
Exploiting Software: How to Break Code (Addison-Wesley Software Security Series)
Extended STL, Volume 1: Collections and Iterators
Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets
Forgotten Realms Campaign Guide, 4th Edition
Foundations of Qt Development
Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries (Microsoft .NET Development Series)
Functional Programming: Practice and Theory
Game Graphics Programming
Golden Fox
Hacking Exposed Windows: Microsoft Windows Security Secrets and Solutions, Third Edition
Hacking Exposed, Sixth Edition: Network Security Secrets And Solutions
Hacking Windows XP (ExtremeTech)
Henry James: Complete Stories 1884-1891 (Library of America)
Henry James: Complete Stories 1864-1874 (Library of America)
Henry James: Complete Stories 1874-1884 (Library of America)
Henry James: Complete Stories 1898-1910 (Library of America)
Henry James: Complete Stories, 1892-1898 (Library of America)
Heretic (The Grail Quest, Book 3)
High Performance Web Sites: Essential Knowledge for Front-End Engineers
How to Ace the Brainteaser Interview
How to Be Evangelical Without Being Conservative
Imperfect C++: Practical Solutions for Real-Life Programming
Implementing Elliptic Curve Cryptography
Inside Microsoft  SQL Server(TM) 2005: The Storage Engine (Solid Quality Learning)
Inside Microsoft SQL Server(TM) 2005: Query Tuning and Optimization
Inside the Revolution: How the Followers of Jihad, Jefferson & Jesus Are Battling to Dominate . . .
Intelligent Data Analysis
Learning Perl, 5th Edition
Learning Python, 3rd Edition
Learning the vi and Vim Editors
Liberal Fascism: The Secret History of the American Left, From Mussolini to the Politics of Meaning
Linux System Programming: Talking Directly to the Kernel and C Library
Linux(R) Debugging and Performance Tuning: Tips and Techniques (Prentice Hall Open Source Software Development Series)
MCITP Self-Paced Training Kit (Exam 70-441): Designing Database Solutions by Using Microsoft  SQL Server(TM) 2005 (Self-Paced Training Kits)
MCTS Self-Paced Training Kit (Exam 70-431): Microsoft SQL Server 2005 Implementation and Maintenance (Pro-Certification)
Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining
Mastering Regular Expressions
Michael Freeman’s Top Digital Photography Tips (A Lark Photography Book)
MicroC OS II: The Real Time Kernel (With CD-ROM)
Microsoft  SQL Server(TM) 2000 High Availability
Microsoft Visual Studio Tips
Microsoft Mobile Development Handbook
More Effective C#: 50 Specific Ways to Improve Your C# (Effective Software Development Series)
Nanny State: How Food Fascists, Teetotaling Do-Gooders, Priggish Moralists, and other Boneheaded Bureaucrats are Turning America into a Nation of Chil
Networking Basics CCNA 1 Companion Guide (Cisco Networking Academy)
Never at Rest: A Biography of Isaac Newton (Cambridge Paperback Library)
No Tech Hacking: A Guide to Social Engineering, Dumpster Diving, and Shoulder Surfing
On Bullshit
Optics (4th Edition)
PC Bios: Improve and Upgrade Your PC’S Computing Power!
Physics of the Impossible: A Scientific Exploration into the World of Phasers, Force Fields, Teleportation, and Time Travel
Point-Counterpoint: Readings in American Government
Power of the Sword
Practical Guide to SysML: The Systems Modeling Language
Practical Internet Security
Practical Packet Analysis: Using Wireshark to Solve Real-World Network Problems
Pro Visual C++ 2005 for C# Developers
Professional Assembly Language (Programmer to Programmer)
Professional Rootkits (Programmer to Programmer)
Professional Visual Studio Extensibility
Programming Erlang: Software for a Concurrent World
Programming Interviews Exposed: Secrets to Landing Your Next Job, 2nd Edition (Programmer to Programmer)
Programming Python
Programming Windows Embedded CE 6.0 Developer Reference, 4th Edition
Programming Windows Security (DevelopMentor Series)
Programming in the Key of C#: A Primer for Aspiring Programmers (Step By Step (Microsoft))
Programming the Microsoft Windows Driver Model, Second Edition
Programming with POSIX(R) Threads (Addison-Wesley Professional Computing Series)
Python Cookbook
Python in a Nutshell (In a Nutshell (O’Reilly))
Quantum Learning: Unleashing the Genius in You
RESTful Web Services
Rage
Reflections on a Theory of Organisms
Rick Sammon’s Exploring the Light: Making the Very Best In-Camera Exposures
SQL Server Forensic Analysis
Secure Programming with Static Analysis (Addison-Wesley Software Security Series)
Skin: The Complete Guide to Digitally Lighting, Photographing, and Retouching Faces and Bodies
Software Estimation: Demystifying the Black Art (Best Practices (Microsoft))
Standard C++ IOStreams and Locales: Advanced Programmer’s Guide and Reference
Strategy: An Introduction to Game Theory, 2nd Edition
Subject To Change: Creating Great Products & Services for an Uncertain World: Adaptive Path on Design (Adaptive Path)
Surely You’re Joking, Mr. Feynman! (Adventures of a Curious Character)
Surreal Numbers
Swords & Circuitry: A Designer’s Guide to Computer Role-Playing Games (Game Development)
System Center Operations Manager 2007 Unleashed
Teach Yourself Ole Programming in 21 Days/Book
The Art of Debugging with GDB, DDD, and Eclipse
The BIOS Companion: The book that doesn’t come with your motherboard!
The Basque Language: A Practical Introduction (The Basque Series)
The Best of 2600: A Hacker Odyssey
The BetterPhoto Guide to Photographing Children (BetterPhoto Series)
The Burning Shore
The C++ Standard Library Extensions: A Tutorial and Reference
The Game Localization Handbook (Game Development Series)
The God Delusion
The Great Terror: A Reassessment
The Haskell Road to Logic, Maths and Programming (Texts in Computing S.)
The Haskell School of Expression: Learning Functional Programming through Multimedia
The New Turing Omnibus: Sixty-Six Excursions in Computer Science
The Notebooks of Henry James
The Princeton Companion to Mathematics
The Ultimate Guide to Video Game Writing and Design
Three Dragon Ante (Dungeon & Dragons)
Time Management for System Administrators
Traditional Wooden Toys: Their History and How to Make Them
Traffic: Why We Drive the Way We Do (and What It Says About Us)
Types and Programming Languages
WPF in Action with Visual Studio 2008
Why Programs Fail, Second Edition: A Guide to Systematic Debugging
Why Programs Fail: A Guide to Systematic Debugging
Windows  via C/C++ (PRO-Developer) (Pro - Developer)
Windows Forensic Analysis Including DVD Toolkit
Windows Home Server: Protect and Simplify your Digital Life
Windows NT File System Internals (OSR Classic Reprints)
Windows PowerShell in Action
Windows XP Hacks
Witches Abroad
Write Faster, Write Better
Yoga Spandakarika: The Sacred Texts at the Origins of Tantra
bash Cookbook: Solutions and Examples for bash Users (Cookbooks (O’Reilly))
The Failure Factory: How Unelected Bureaucrats, Liberal Democrats, and Big Government Republicans Are Undermining America’s Security and Leading Us to War

- Dmitry Vostokov @ DumpAnalysis.org -

Now some pictures from Citrix CDF traces.

Deep waters of The Sea of Traces:

Considering software tracing as narrative no wonder one day I discovered the vast Library of Software Logs in the sea above:

- Dmitry Vostokov @ DumpAnalysis.org -

One of the questions that is often asked is about exception addresses in application and system event logs. For example, we have this typical log entry:

"The instruction at "0x77ca8fa7" referenced memory address "0x00000000". The memory could not be read."

Suppose the dump is not available. Can we find the function address to look in our problem database?

My answer here is usually the following:

Even if the application is no longer running we can either noninvasively attach a debugger to it or get a user dump of it and later find the nearest address using ln WinDbg command (remember to apply correct symbols first, see windbg.org):

0:000> ln 77ca8fa7
(77ca8f91)   msvcrt!wcscpy+0×16   |  (77ca8fd6)   msvcrt!wcspbrk

Usually applications crash inside functions and not at their entry addresses, so we pay attention to wcscpy function because it has the offset +0×16.

Note: on Vista and W2K8 due to ASLR, system DLLs could be at different addresses if we take the dump of or attach to a different running process instance.

- Dmitry Vostokov @ DumpAnalysis.org -

I was suddenly enlightened by the unification of software traces with memory dumps and it came to me that DATA is simply Dump Analysis + Trace Analysis. It is commutative with TADA, the sound of accomplishment (tada.wav in Windows \ Media folder).

This can’t be a coincidence, can it?

- Dmitry Vostokov @ DumpAnalysis.org -

What is a software trace from a mathematical standpoint? Before any software writes its trace data, it assembles it in memory. Therefore, generally, a software trace is a linear ordered sequence of specifically prepared memory fragments (trace statements):

(ts₁, ts₂, …, ts_n) 

where every ts_i is a sequence of bits, bytes or other discrete units (see the definition of a memory dump):

(s₁₁, s₁₂, …, s_1k, s₂₁, s₂₂, …, s_2l, …, …, …, s_n1, s_n2, …, s_nm)

These trace statements can also be minidumps, selected regions of memory space. In the limit, if every ts_i is a full memory snapshot saved at an instant of time (t_i) we have a sequence of memory dumps:

(m_t1, m_t2, …, m_tn) 

Like with memory dump analysis we need symbol files to interpret saved memory fragments unless they were already interpreted during their construction. For example, traces written according ETW specification (Event Tracing for Windows), need TMF files (Trace Message Format) for their interpretation and viewing. Usually these files are generated from PDB files and therefore we have this correspondence:

memory dump file -> software trace file

PDB file -> TMF file 

- Dmitry Vostokov @ TraceAnalysis.org -

Do you remember memuons¹, the indivisible entities of memory? Their study is the domain of the new science called memuonics². According to the so called memophysical principle³,we have particle interpretation of memuons. This is called classical memuonics with classical memory field theory where memuons are “quanta” of memory. We can also ”quantize” memory fields and get quantum memory field theories where memuons are created and annihilated.

(1) The notion of memuons first appeared in the philosophy of memoidealism.

(2) Please don’t confuse memuonics with memiotics. The latter is computer memory semiotics.

(3) Memophysical principle - theories of memory-based universe need to take into account the current mainstream sciences including physics.

- Dmitry Vostokov @ DumpAnalysis.org -

“A trace is a narrative, the story of a computation.”

Dmitry Vostokov, Software Tracing and Logging: Architecture, Design, Implementation and Analysis Patterns

- Dmitry Vostokov @ TraceAnalysis.org -

Found today this book while browsing Amazon:

The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System

Intrigued, I searched for its table of contents and found the author’s site:

Book TOC

Looks enough comprehensive so I pre-ordered the book and plan to write a review later from windows internals and memory dump analysis perspective.

- Dmitry Vostokov @ DumpAnalysis.org -

This is a stack trace (backtrace) where all finctions are removed and only modules are left. Useful for depicting component dependencies. Here is an example:

0: kd> kc 100
nt!KiSwapContext
nt!KiSwapThread
nt!KeDelayExecutionThread
3rdPartyAVDriver
3rdPartyAVDriver
3rdPartyAVDriver
3rdPartyAVDriver
nt!IofCallDriver
DriverA!Dispatch
DriverA!KUI_dispatch
nt!IofCallDriver
DriverB!PassThrough
DriverB!Dispatch
nt!IofCallDriver
DriverC!LowerPassThrough
DriverC
DriverC
DriverC
DriverC!DispatchPassThrough
nt!IofCallDriver
nt!IopParseDevice
nt!ObpLookupObjectName
nt!ObOpenObjectByName
nt!IopCreateFile
nt!IoCreateFile
nt!NtOpenFile
nt!KiFastCallEntry
ntdll!KiFastSystemCallRet
ntdll!NtOpenFile
ntdll!LdrpCreateDllSection
ntdll!LdrpMapDll
ntdll!LdrpLoadDll
ntdll!LdrLoadDll
kernel32!LoadLibraryExW
kernel32!LoadLibraryW
ntdll!LdrpCallInitRoutine
ntdll!LdrpRunInitializeRoutines
ntdll!LdrpLoadDll
ntdll!LdrLoadDll
kernel32!LoadLibraryExW
kernel32!LoadLibraryW
USER32!LoadAppDlls
USER32!ClientThreadSetup
USER32!__ClientThreadSetup
ntdll!KiUserCallbackDispatcher
nt!KiCallUserMode
nt!KeUserModeCallback
win32k!xxxClientThreadSetup
win32k!xxxCreateThreadInfo
win32k!UserThreadCallout
win32k!W32pThreadCallout
nt!PsConvertToGuiThread
nt!KiBBTUnexpectedRange

Collapsed stack trace is much simpler to grasp: 

nt
3rdPartyAVDriver
nt
DriverA
nt
DriverB
nt
DriverC
nt
ntdll
kernel32
ntdll
kernel32
USER32
ntdll
nt
win32k
nt

- Dmitry Vostokov @ DumpAnalysis.org -

Inspired by Penrose tensor notation encountered in The Road to Reality book and Feynman diagrams I’d like to introduce Visual Dump Objects (VDO) graphical notation to depict and communicate memory dump analysis patterns, their combinations and analysis results. Let’s look at some basic visual objects (shown in handwriting).

1. A thread:

   or   

2. A function:

3. A module:

4. A thread running through functions, modules or both (stack trace). Optional arrowhead can indicate stack trace direction:

  or    or  

Threads running through modules depict collapsed stack traces.

5. A blocked thread:

An example of 3 threads blocked by another thread (an arrowhead can disambiguate the direction of the waiting chain):

6. A spiking thread (colors are encouraged in VDO notation):

   or   

7. Space boundary between user land and kernel land:

Here is an example of the thread spiking in kernel space:

or with modules from stack trace:

More notation to come very soon.

- Dmitry Vostokov @ DumpAnalysis.org -

Instead of publishing another philosophical treatise ”On …”, OpenTask, iterative and incremental publisher, plans to release my collection of bugtations in somewhat extended version by the end of this summer:  

On Debugging: Bugtations and Other Humorous Quotations (ISBN: 978-1906717285)

The book also includes short biographical notes, commentaries and relevant explanations. Hope you would enjoy it. 

- Dmitry Vostokov @ DumpAnalysis.org -