Software Diagnostics Library

DumpAnalysis.org has changed its book store to Amazon aStore to incorporate all published OpenTask books, magazines and notebooks:

Here is the direct link:

Dump Analysis Portal Store

The screenshot:

 - Dmitry Vostokov @ DumpAnalysis.org -

I’ve been a member of Amazon Associates program since June 2007, providing links to books on my various blogs. Visitors did almost 20,000 clicks and bought almost 1,000 books (although not always ones that I recommended). Here is the list of bought or pre-ordered titles sorted by popularity:

Memory Dump Analysis Anthology, Volume 1
Advanced Windows Debugging (The Addison-Wesley Microsoft Technology Series)
Windows Debugging: Practical Foundations
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Windows Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (PRO-Developer)
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The IDA Pro Book: The Unofficial Guide to the World’s Most Popular Disassembler
Windows via C/C++ (Pro - Developer)
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
Windows Sysinternals Administrator’s Reference (Inside Out)
Microsoft Windows Internals (4th Edition): Microsoft Windows Server 2003, Windows XP, and Windows 2000
Running Xen: A Hands-On Guide to the Art of Virtualization
The Definitive Guide to the Xen Hypervisor (Prentice Hall Open Source Software Development Series)
Baby Turing
Inside the Machine: An Illustrated Introduction to Microprocessors and Computer Architecture
Reversing: Secrets of Reverse Engineering
The Developer’s Guide to Debugging
The Old New Thing: Practical Development Throughout the Evolution of Windows
Debugging Microsoft .NET 2.0 Applications
Developing Drivers with the Windows  Driver Foundation (Pro Developer)
The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System
Apache Server 2.0: The Complete Reference
Becoming a Technical Leader: An Organic Problem-Solving Approach
C++ Primer Plus (5th Edition)
DLL List Landscape: The Art from Computer Memory Space
Debugging by Thinking: A Multidisciplinary Approach (HP Technologies)
Designing Storage Area Networks: A Practical Reference for Implementing Fibre Channel and IP SANs (2nd Edition)
Hacking: The Art of Exploitation, 2nd Edition
Inside the C++ Object Model
Linux Kernel Development (2nd Edition) (Novell Press)
MCITP Self-Paced Training Kit (Exams 70-640, 70-642, 70-643, 70-647): Windows Server 2008 Enterprise Administrator Core Requirements
Microsoft Visual C# 2008 Step by Step
Programming Interviews Exposed: Secrets to Landing Your Next Job (Programmer to Programmer)
Programming Language Pragmatics, Second Edition
Python Essential Reference (3rd Edition) (Developer’s Library)
Reverse Engineering Code with IDA Pro
Ring Bearer, The
Rootkits: Subverting the Windows Kernel (Addison-Wesley Software Security Series)
Strong Women, Strong Bones, Updated
The Back of the Napkin: Solving Problems and Selling Ideas with Pictures
Windows NT/2000 Native API Reference (Circle)
Writing Secure Code for Windows Vista (Pro - Step By Step Developer)
A Time to Die
Accelerated Learning for the 21st Century: The Six-Step Plan to Unlock Your Master-Mind
Administering Windows Vista Security: The Big Surprises
Advanced Programming in the UNIX(R) Environment (2nd Edition) (Addison-Wesley Professional Computing Series)
Advanced Topics in Types and Programming Languages
All the Mathematics You Missed: But Need to Know for Graduate School
An Introduction to Lambda Calculi for Computer Scientists
Asterisk: The Future of Telephony
BIOS Disassembly Ninjutsu Uncovered (Uncovered series)
Basic Abstract Algebra
Basic Category Theory for Computer Scientists (Foundations of Computing)
Bit by Bit: An Illustrated History of Computers
Breaking Through the BIOS Barrier: The Definitive BIOS Optimization Guide for PCs
C++ GUI Programming with Qt4 (2nd Edition) (Prentice Hall Open Source Software Development Series)
C++ Iostreams Handbook
C++ Template Metaprogramming: Concepts, Tools, and Techniques from Boost and Beyond (C++ In-Depth Series)
C: A Reference Manual (5th Edition)
CLR via C#, Second Edition (Pro Developer)
Code Complete: A Practical Handbook of Software Construction
Code Craft: The Practice of Writing Excellent Code
Code: The Hidden Language of Computer Hardware and Software
Compilers: Principles, Techniques, and Tools (2nd Edition)
Complete Digital Photography, Fourth Edition (Graphics Series)
Computer Repair with Diagnostic Flowcharts: Troubleshooting PC Hardware Problems from Boot Failure to Poor Performance, Revised Edition
Computer Science Made Simple: Learn how hardware and software work– and how to make them work for you! (Made Simple)
Concepts, Techniques, and Models of Computer Programming
Cross-Platform Development in C++: Building Mac OS X, Linux, and Windows Applications
Data Mining and Knowledge Discovery Handbook
Debugging
Debugging Applications: Microsoft (Dv-Mps Programming)
Debugging Windows Programs: Strategies, Tools, and Techniques for Visual C++ Programmers (The DevelopMentor Series)
Developing Drivers with the Windows  Driver Foundation
Domain-Specific Development with Visual Studio DSL Tools (Microsoft .NET Development Series)
Electronic Data Interchange in Finance and Accounting
Exceptional C++: 47 Engineering Puzzles, Programming Problems, and Solutions (C++ In-Depth Series)
Expert F# (Expert’s Voice in .Net)
Exploiting Software: How to Break Code (Addison-Wesley Software Security Series)
Extended STL, Volume 1: Collections and Iterators
Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets
Forgotten Realms Campaign Guide, 4th Edition
Foundations of Qt Development
Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries (Microsoft .NET Development Series)
Functional Programming: Practice and Theory
Game Graphics Programming
Golden Fox
Hacking Exposed Windows: Microsoft Windows Security Secrets and Solutions, Third Edition
Hacking Exposed, Sixth Edition: Network Security Secrets And Solutions
Hacking Windows XP (ExtremeTech)
Henry James: Complete Stories 1884-1891 (Library of America)
Henry James: Complete Stories 1864-1874 (Library of America)
Henry James: Complete Stories 1874-1884 (Library of America)
Henry James: Complete Stories 1898-1910 (Library of America)
Henry James: Complete Stories, 1892-1898 (Library of America)
Heretic (The Grail Quest, Book 3)
High Performance Web Sites: Essential Knowledge for Front-End Engineers
How to Ace the Brainteaser Interview
How to Be Evangelical Without Being Conservative
Imperfect C++: Practical Solutions for Real-Life Programming
Implementing Elliptic Curve Cryptography
Inside Microsoft  SQL Server(TM) 2005: The Storage Engine (Solid Quality Learning)
Inside Microsoft SQL Server(TM) 2005: Query Tuning and Optimization
Inside the Revolution: How the Followers of Jihad, Jefferson & Jesus Are Battling to Dominate . . .
Intelligent Data Analysis
Learning Perl, 5th Edition
Learning Python, 3rd Edition
Learning the vi and Vim Editors
Liberal Fascism: The Secret History of the American Left, From Mussolini to the Politics of Meaning
Linux System Programming: Talking Directly to the Kernel and C Library
Linux(R) Debugging and Performance Tuning: Tips and Techniques (Prentice Hall Open Source Software Development Series)
MCITP Self-Paced Training Kit (Exam 70-441): Designing Database Solutions by Using Microsoft  SQL Server(TM) 2005 (Self-Paced Training Kits)
MCTS Self-Paced Training Kit (Exam 70-431): Microsoft SQL Server 2005 Implementation and Maintenance (Pro-Certification)
Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining
Mastering Regular Expressions
Michael Freeman’s Top Digital Photography Tips (A Lark Photography Book)
MicroC OS II: The Real Time Kernel (With CD-ROM)
Microsoft  SQL Server(TM) 2000 High Availability
Microsoft Visual Studio Tips
Microsoft Mobile Development Handbook
More Effective C#: 50 Specific Ways to Improve Your C# (Effective Software Development Series)
Nanny State: How Food Fascists, Teetotaling Do-Gooders, Priggish Moralists, and other Boneheaded Bureaucrats are Turning America into a Nation of Chil
Networking Basics CCNA 1 Companion Guide (Cisco Networking Academy)
Never at Rest: A Biography of Isaac Newton (Cambridge Paperback Library)
No Tech Hacking: A Guide to Social Engineering, Dumpster Diving, and Shoulder Surfing
On Bullshit
Optics (4th Edition)
PC Bios: Improve and Upgrade Your PC’S Computing Power!
Physics of the Impossible: A Scientific Exploration into the World of Phasers, Force Fields, Teleportation, and Time Travel
Point-Counterpoint: Readings in American Government
Power of the Sword
Practical Guide to SysML: The Systems Modeling Language
Practical Internet Security
Practical Packet Analysis: Using Wireshark to Solve Real-World Network Problems
Pro Visual C++ 2005 for C# Developers
Professional Assembly Language (Programmer to Programmer)
Professional Rootkits (Programmer to Programmer)
Professional Visual Studio Extensibility
Programming Erlang: Software for a Concurrent World
Programming Interviews Exposed: Secrets to Landing Your Next Job, 2nd Edition (Programmer to Programmer)
Programming Python
Programming Windows Embedded CE 6.0 Developer Reference, 4th Edition
Programming Windows Security (DevelopMentor Series)
Programming in the Key of C#: A Primer for Aspiring Programmers (Step By Step (Microsoft))
Programming the Microsoft Windows Driver Model, Second Edition
Programming with POSIX(R) Threads (Addison-Wesley Professional Computing Series)
Python Cookbook
Python in a Nutshell (In a Nutshell (O’Reilly))
Quantum Learning: Unleashing the Genius in You
RESTful Web Services
Rage
Reflections on a Theory of Organisms
Rick Sammon’s Exploring the Light: Making the Very Best In-Camera Exposures
SQL Server Forensic Analysis
Secure Programming with Static Analysis (Addison-Wesley Software Security Series)
Skin: The Complete Guide to Digitally Lighting, Photographing, and Retouching Faces and Bodies
Software Estimation: Demystifying the Black Art (Best Practices (Microsoft))
Standard C++ IOStreams and Locales: Advanced Programmer’s Guide and Reference
Strategy: An Introduction to Game Theory, 2nd Edition
Subject To Change: Creating Great Products & Services for an Uncertain World: Adaptive Path on Design (Adaptive Path)
Surely You’re Joking, Mr. Feynman! (Adventures of a Curious Character)
Surreal Numbers
Swords & Circuitry: A Designer’s Guide to Computer Role-Playing Games (Game Development)
System Center Operations Manager 2007 Unleashed
Teach Yourself Ole Programming in 21 Days/Book
The Art of Debugging with GDB, DDD, and Eclipse
The BIOS Companion: The book that doesn’t come with your motherboard!
The Basque Language: A Practical Introduction (The Basque Series)
The Best of 2600: A Hacker Odyssey
The BetterPhoto Guide to Photographing Children (BetterPhoto Series)
The Burning Shore
The C++ Standard Library Extensions: A Tutorial and Reference
The Game Localization Handbook (Game Development Series)
The God Delusion
The Great Terror: A Reassessment
The Haskell Road to Logic, Maths and Programming (Texts in Computing S.)
The Haskell School of Expression: Learning Functional Programming through Multimedia
The New Turing Omnibus: Sixty-Six Excursions in Computer Science
The Notebooks of Henry James
The Princeton Companion to Mathematics
The Ultimate Guide to Video Game Writing and Design
Three Dragon Ante (Dungeon & Dragons)
Time Management for System Administrators
Traditional Wooden Toys: Their History and How to Make Them
Traffic: Why We Drive the Way We Do (and What It Says About Us)
Types and Programming Languages
WPF in Action with Visual Studio 2008
Why Programs Fail, Second Edition: A Guide to Systematic Debugging
Why Programs Fail: A Guide to Systematic Debugging
Windows  via C/C++ (PRO-Developer) (Pro - Developer)
Windows Forensic Analysis Including DVD Toolkit
Windows Home Server: Protect and Simplify your Digital Life
Windows NT File System Internals (OSR Classic Reprints)
Windows PowerShell in Action
Windows XP Hacks
Witches Abroad
Write Faster, Write Better
Yoga Spandakarika: The Sacred Texts at the Origins of Tantra
bash Cookbook: Solutions and Examples for bash Users (Cookbooks (O’Reilly))
The Failure Factory: How Unelected Bureaucrats, Liberal Democrats, and Big Government Republicans Are Undermining America’s Security and Leading Us to War

- Dmitry Vostokov @ DumpAnalysis.org -

What is a software trace from a mathematical standpoint? Before any software writes its trace data, it assembles it in memory. Therefore, generally, a software trace is a linear ordered sequence of specifically prepared memory fragments (trace statements):

(ts₁, ts₂, …, ts_n) 

where every ts_i is a sequence of bits, bytes or other discrete units (see the definition of a memory dump):

(s₁₁, s₁₂, …, s_1k, s₂₁, s₂₂, …, s_2l, …, …, …, s_n1, s_n2, …, s_nm)

These trace statements can also be minidumps, selected regions of memory space. In the limit, if every ts_i is a full memory snapshot saved at an instant of time (t_i) we have a sequence of memory dumps:

(m_t1, m_t2, …, m_tn) 

Like with memory dump analysis we need symbol files to interpret saved memory fragments unless they were already interpreted during their construction. For example, traces written according ETW specification (Event Tracing for Windows), need TMF files (Trace Message Format) for their interpretation and viewing. Usually these files are generated from PDB files and therefore we have this correspondence:

memory dump file -> software trace file

PDB file -> TMF file 

- Dmitry Vostokov @ TraceAnalysis.org -

Do you remember memuons¹, the indivisible entities of memory? Their study is the domain of the new science called memuonics². According to the so called memophysical principle³,we have particle interpretation of memuons. This is called classical memuonics with classical memory field theory where memuons are “quanta” of memory. We can also ”quantize” memory fields and get quantum memory field theories where memuons are created and annihilated.

(1) The notion of memuons first appeared in the philosophy of memoidealism.

(2) Please don’t confuse memuonics with memiotics. The latter is computer memory semiotics.

(3) Memophysical principle - theories of memory-based universe need to take into account the current mainstream sciences including physics.

- Dmitry Vostokov @ DumpAnalysis.org -

Inspired by Penrose tensor notation encountered in The Road to Reality book and Feynman diagrams I’d like to introduce Visual Dump Objects (VDO) graphical notation to depict and communicate memory dump analysis patterns, their combinations and analysis results. Let’s look at some basic visual objects (shown in handwriting).

1. A thread:

   or   

2. A function:

3. A module:

4. A thread running through functions, modules or both (stack trace). Optional arrowhead can indicate stack trace direction:

  or    or  

Threads running through modules depict collapsed stack traces.

5. A blocked thread:

An example of 3 threads blocked by another thread (an arrowhead can disambiguate the direction of the waiting chain):

6. A spiking thread (colors are encouraged in VDO notation):

   or   

7. Space boundary between user land and kernel land:

Here is an example of the thread spiking in kernel space:

or with modules from stack trace:

More notation to come very soon.

- Dmitry Vostokov @ DumpAnalysis.org -

Instead of publishing another philosophical treatise ”On …”, OpenTask, iterative and incremental publisher, plans to release my collection of bugtations in somewhat extended version by the end of this summer:  

On Debugging: Bugtations and Other Humorous Quotations (ISBN: 978-1906717285)

The book also includes short biographical notes, commentaries and relevant explanations. Hope you would enjoy it. 

- Dmitry Vostokov @ DumpAnalysis.org -

This is a forthcoming book to be released next year after we finally publish DebugWare book by the end of this summer:

Software Tracing and Logging: Architecture, Design, Implementation and Analysis Patterns (ISBN: 978-1906717728)

I have already begun working on it in the background. The scope of DebugWare book is too wide to cover tracing and logging in great detail not to mention the very important subject of software trace analysis.

- Dmitry Vostokov @ TraceAnalysis.org -

Last weekend I came across this book in a local bookshop:

Patterns Of The Earth

I immediately recalled all beautiful patterns I have seen and collected from visualized computer memory maps. I plan to include many images in a full-color book this summer:

Patterns of Computer Memory (ISBN: 978-1906717711)

This book should be essential for creative computer engineers and inquiring debugging minds!

- Dmitry Vostokov @ DumpAnalysis.org -

This post number is 1,000. Wordpress admin panel reports that I have written 999 (excluding this one):

Of course, I wrote my 1,000^th blog post some time ago if I count all my 7 blogs but this is 1,000^th post for just this blog.

To celebrate this event, I’m posting a picture of troubleshooting and debugging labyrinth resting on a notion of universal memory dumps that are observational snapshots and include both memory and various traces we collect to resolve problems.

This picture shows how we arrive to problem resolution. For example:  

I’m also working on another picture, called T&D Tangram

- Dmitry Vostokov @ DumpAnalysis.org -

OpenTask, iterative and incremental publisher, plans to release a guide for general computer users explaining crashes and hangs on personal computers ranging from mobile phones, netbooks and laptops to desktops and servers:

My Computer Crashes and Freezes: A Non-technical Guide to Software and Hardware Errors (ISBN: 978-19067171704)

- Dmitry Vostokov @ DumpAnalysis.org -

I recently became interested in who else has the same surname. Vostokov means Mr. East, east - ost (Ger.) - vostok (Rus.). Here is the brief list of people with links I’m going to expand periodically.

Alexander Vostokov - One of the first Russian philologists

Yevgeniy Vostokov - Composer

Sergei Vostokov - Mathematician

Viktor Vostokov - Author. “Secrets of East Healers”, “Lessons of Tibetan Medicine: The Art of Relaxation” and other books in Russian

Eugeny Vostokov - Artist

More to come.

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced memory dump analysis certification will have x86/x64-based tracks for Windows and Unix (including Linux / FreeBSD / Mac OS X). Each track consists of 3 exams, each having its own set of requirements and scope:

• Fundamentals of Memory Dump Analysis
• Intermediate Memory Dump Analysis
• Advanced Memory Dump Analysis

More information will be available later. The initiative is supported by OpenTask.

- Dmitry Vostokov @ DumpAnalysis.org -

As soon as I wrote my review of the 2nd edition I found out that the 3rd edition was recently published and immediately bought it. I intend to read it from cover to cover again and publish my notes and comments in my reading notebook on Software Generalist blog. The new edition is also bundled with a companion CD.

Programming Language Pragmatics, Third Edition

Hope in one of subsequent editions the author includes my Riemann Programming Language :-)

- Dmitry Vostokov @ DumpAnalysis.org -

This is the enhanced version of Dump2Wave technology that allows to transform computational operations into audible artifacts.

Computational processes and threads are fiber bundled with native memory visualization techniques to create audio and visual images of powerful memory topoi. This opens the new era in music. The closure of analog -> digital -> analog enables visualization and auralization of finite and infinite (transfinite) digital data.

Stay tuned! More on this later…

- Dmitry Vostokov @ DumpAnalysis.org -

The community of Citrix and Microsoft Terminal Services / Virtualization meets again in Dublin, Ireland:

Event: PubForum 2009 Dublin - just one this year
Date: 5 to 7 June 2009
Where: Dublin, Ireland
Event Location: Camden Court Hotel, Dublin
Sessions: 20
Master Classes: 6
Visitors: 75 - 80
Attendee Fee: just 199 EUR

Agenda of the Event

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Management Bits and Tips blog:

Flattening My Management

Strategic and Tactical Personal Learning

Customer Relationship Martyr 

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Literate Scientist blog:

Young Stalin

General Chemistry

Evolution: The First Four Billion Years

More Than a Theory

Understanding the Infinite

Conceptual Mathematics

The Coming of the Third Reich

Comrades

- Dmitry Vostokov @ DumpAnalysis.org -

After coming back to engineering I decided to expand the domain of my research and start the new series of posts called Trace Analysis Patterns. In addition to Citrix CDF / Microsoft ETW traces I plan to cover other variants based on my extensive software engineering background in the past where I used tracing in software products ranging from soft multi-platform real-time systems to static code analysis tools. Connection with memory dump analysis will be covered too because sometimes the combination of static and dynamic data leads to interesting observations and helps to troubleshoot and resolve customer problems especially when not all data can be collected dynamically.

In fact, stack traces and their collections are specializations of the more general traces. Another example is historical information in memory dump files especially when it is somehow timestamped.  

In this part I start with the obvious and to some extent the trivial pattern called Periodic Error. This is an error or a status value that is observed periodically many times:

No     PID  TID   Date      Time         Statement
[...]
664957 1788 22504 4/23/2009 17:59:14.600 MyClass::Initialize: Cannot open connection “Client ID: 310″, status=5  
[…]
668834 1788 19868 4/23/2009 19:11:52.979 MyClass::Initialize: Cannot open connection “Client ID: 612″, status=5 
[…]

or 

No     PID  TID   Date      Time         Statement
[...] 
202314 1788 19128 4/21/2009 16:03:46.861 HandleDataLevel: Error 12005 Getting Mask
[…]
347653 1788 17812 4/22/2009 13:26:00.735 HandleDataLevel: Error 12005 Getting Mask
[…]

Here single trace entries can be isolated from the trace and studied in detail. 

Be aware though that some modules might report periodic errors that are false positive, in the sense, that they are expected as a part of implementation details, for example, when a function returns an error to indicate that bigger buffer is required or to estimate its size for a subsequent call. It merits its own pattern name and I come to it next time with more examples.

I also created a page where I’ll will be adding all tracing patterns:

Trace Analysis Patterns   

- Dmitry Vostokov @ TraceAnalysis.org -

DumpAnalysis.org announces forthcoming 2011 - 2020 as The Debugging Decade.

Q&A

Q. Why 2011 - 2020?

A. The main reason is the fact that 2009 is The Year of Debugging and 2010 is The Year of Dump Analysis. This naturally extends to a decade.

Q. Do you plan The Debugging Century?

A. Yes, I do. Details will be announced later.

- Dmitry Vostokov @ DumpAnalysis.org -