Archive for May, 2009

New Portal Store

Sunday, May 31st, 2009

DumpAnalysis.org has changed its book store to Amazon aStore to incorporate all published OpenTask books, magazines and notebooks:

Here is the direct link:

Dump Analysis Portal Store

The screenshot:

 - Dmitry Vostokov @ DumpAnalysis.org -

2 Years of Amazon Associate

Sunday, May 31st, 2009

I’ve been a member of Amazon Associates program since June 2007, providing links to books on my various blogs. Visitors did almost 20,000 clicks and bought almost 1,000 books (although not always ones that I recommended). Here is the list of bought or pre-ordered titles sorted by popularity:

Memory Dump Analysis Anthology, Volume 1
Advanced Windows Debugging (The Addison-Wesley Microsoft Technology Series)
Windows Debugging: Practical Foundations
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Windows Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (PRO-Developer)
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The IDA Pro Book: The Unofficial Guide to the World’s Most Popular Disassembler
Windows via C/C++ (Pro - Developer)
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
Windows Sysinternals Administrator’s Reference (Inside Out)
Microsoft Windows Internals (4th Edition): Microsoft Windows Server 2003, Windows XP, and Windows 2000
Running Xen: A Hands-On Guide to the Art of Virtualization
The Definitive Guide to the Xen Hypervisor (Prentice Hall Open Source Software Development Series)
Baby Turing
Inside the Machine: An Illustrated Introduction to Microprocessors and Computer Architecture
Reversing: Secrets of Reverse Engineering
The Developer’s Guide to Debugging
The Old New Thing: Practical Development Throughout the Evolution of Windows
Debugging Microsoft .NET 2.0 Applications
Developing Drivers with the Windows  Driver Foundation (Pro Developer)
The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System
Apache Server 2.0: The Complete Reference
Becoming a Technical Leader: An Organic Problem-Solving Approach
C++ Primer Plus (5th Edition)
DLL List Landscape: The Art from Computer Memory Space
Debugging by Thinking: A Multidisciplinary Approach (HP Technologies)
Designing Storage Area Networks: A Practical Reference for Implementing Fibre Channel and IP SANs (2nd Edition)
Hacking: The Art of Exploitation, 2nd Edition
Inside the C++ Object Model
Linux Kernel Development (2nd Edition) (Novell Press)
MCITP Self-Paced Training Kit (Exams 70-640, 70-642, 70-643, 70-647): Windows Server 2008 Enterprise Administrator Core Requirements
Microsoft Visual C# 2008 Step by Step
Programming Interviews Exposed: Secrets to Landing Your Next Job (Programmer to Programmer)
Programming Language Pragmatics, Second Edition
Python Essential Reference (3rd Edition) (Developer’s Library)
Reverse Engineering Code with IDA Pro
Ring Bearer, The
Rootkits: Subverting the Windows Kernel (Addison-Wesley Software Security Series)
Strong Women, Strong Bones, Updated
The Back of the Napkin: Solving Problems and Selling Ideas with Pictures
Windows NT/2000 Native API Reference (Circle)
Writing Secure Code for Windows Vista (Pro - Step By Step Developer)
A Time to Die
Accelerated Learning for the 21st Century: The Six-Step Plan to Unlock Your Master-Mind
Administering Windows Vista Security: The Big Surprises
Advanced Programming in the UNIX(R) Environment (2nd Edition) (Addison-Wesley Professional Computing Series)
Advanced Topics in Types and Programming Languages
All the Mathematics You Missed: But Need to Know for Graduate School
An Introduction to Lambda Calculi for Computer Scientists
Asterisk: The Future of Telephony
BIOS Disassembly Ninjutsu Uncovered (Uncovered series)
Basic Abstract Algebra
Basic Category Theory for Computer Scientists (Foundations of Computing)
Bit by Bit: An Illustrated History of Computers
Breaking Through the BIOS Barrier: The Definitive BIOS Optimization Guide for PCs
C++ GUI Programming with Qt4 (2nd Edition) (Prentice Hall Open Source Software Development Series)
C++ Iostreams Handbook
C++ Template Metaprogramming: Concepts, Tools, and Techniques from Boost and Beyond (C++ In-Depth Series)
C: A Reference Manual (5th Edition)
CLR via C#, Second Edition (Pro Developer)
Code Complete: A Practical Handbook of Software Construction
Code Craft: The Practice of Writing Excellent Code
Code: The Hidden Language of Computer Hardware and Software
Compilers: Principles, Techniques, and Tools (2nd Edition)
Complete Digital Photography, Fourth Edition (Graphics Series)
Computer Repair with Diagnostic Flowcharts: Troubleshooting PC Hardware Problems from Boot Failure to Poor Performance, Revised Edition
Computer Science Made Simple: Learn how hardware and software work– and how to make them work for you! (Made Simple)
Concepts, Techniques, and Models of Computer Programming
Cross-Platform Development in C++: Building Mac OS X, Linux, and Windows Applications
Data Mining and Knowledge Discovery Handbook
Debugging
Debugging Applications: Microsoft (Dv-Mps Programming)
Debugging Windows Programs: Strategies, Tools, and Techniques for Visual C++ Programmers (The DevelopMentor Series)
Developing Drivers with the Windows  Driver Foundation
Domain-Specific Development with Visual Studio DSL Tools (Microsoft .NET Development Series)
Electronic Data Interchange in Finance and Accounting
Exceptional C++: 47 Engineering Puzzles, Programming Problems, and Solutions (C++ In-Depth Series)
Expert F# (Expert’s Voice in .Net)
Exploiting Software: How to Break Code (Addison-Wesley Software Security Series)
Extended STL, Volume 1: Collections and Iterators
Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets
Forgotten Realms Campaign Guide, 4th Edition
Foundations of Qt Development
Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries (Microsoft .NET Development Series)
Functional Programming: Practice and Theory
Game Graphics Programming
Golden Fox
Hacking Exposed Windows: Microsoft Windows Security Secrets and Solutions, Third Edition
Hacking Exposed, Sixth Edition: Network Security Secrets And Solutions
Hacking Windows XP (ExtremeTech)
Henry James: Complete Stories 1884-1891 (Library of America)
Henry James: Complete Stories 1864-1874 (Library of America)
Henry James: Complete Stories 1874-1884 (Library of America)
Henry James: Complete Stories 1898-1910 (Library of America)
Henry James: Complete Stories, 1892-1898 (Library of America)
Heretic (The Grail Quest, Book 3)
High Performance Web Sites: Essential Knowledge for Front-End Engineers
How to Ace the Brainteaser Interview
How to Be Evangelical Without Being Conservative
Imperfect C++: Practical Solutions for Real-Life Programming
Implementing Elliptic Curve Cryptography
Inside Microsoft  SQL Server(TM) 2005: The Storage Engine (Solid Quality Learning)
Inside Microsoft SQL Server(TM) 2005: Query Tuning and Optimization
Inside the Revolution: How the Followers of Jihad, Jefferson & Jesus Are Battling to Dominate . . .
Intelligent Data Analysis
Learning Perl, 5th Edition
Learning Python, 3rd Edition
Learning the vi and Vim Editors
Liberal Fascism: The Secret History of the American Left, From Mussolini to the Politics of Meaning
Linux System Programming: Talking Directly to the Kernel and C Library
Linux(R) Debugging and Performance Tuning: Tips and Techniques (Prentice Hall Open Source Software Development Series)
MCITP Self-Paced Training Kit (Exam 70-441): Designing Database Solutions by Using Microsoft  SQL Server(TM) 2005 (Self-Paced Training Kits)
MCTS Self-Paced Training Kit (Exam 70-431): Microsoft SQL Server 2005 Implementation and Maintenance (Pro-Certification)
Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining
Mastering Regular Expressions
Michael Freeman’s Top Digital Photography Tips (A Lark Photography Book)
MicroC OS II: The Real Time Kernel (With CD-ROM)
Microsoft  SQL Server(TM) 2000 High Availability
Microsoft Visual Studio Tips
Microsoft Mobile Development Handbook
More Effective C#: 50 Specific Ways to Improve Your C# (Effective Software Development Series)
Nanny State: How Food Fascists, Teetotaling Do-Gooders, Priggish Moralists, and other Boneheaded Bureaucrats are Turning America into a Nation of Chil
Networking Basics CCNA 1 Companion Guide (Cisco Networking Academy)
Never at Rest: A Biography of Isaac Newton (Cambridge Paperback Library)
No Tech Hacking: A Guide to Social Engineering, Dumpster Diving, and Shoulder Surfing
On Bullshit
Optics (4th Edition)
PC Bios: Improve and Upgrade Your PC’S Computing Power!
Physics of the Impossible: A Scientific Exploration into the World of Phasers, Force Fields, Teleportation, and Time Travel
Point-Counterpoint: Readings in American Government
Power of the Sword
Practical Guide to SysML: The Systems Modeling Language
Practical Internet Security
Practical Packet Analysis: Using Wireshark to Solve Real-World Network Problems
Pro Visual C++ 2005 for C# Developers
Professional Assembly Language (Programmer to Programmer)
Professional Rootkits (Programmer to Programmer)
Professional Visual Studio Extensibility
Programming Erlang: Software for a Concurrent World
Programming Interviews Exposed: Secrets to Landing Your Next Job, 2nd Edition (Programmer to Programmer)
Programming Python
Programming Windows Embedded CE 6.0 Developer Reference, 4th Edition
Programming Windows Security (DevelopMentor Series)
Programming in the Key of C#: A Primer for Aspiring Programmers (Step By Step (Microsoft))
Programming the Microsoft Windows Driver Model, Second Edition
Programming with POSIX(R) Threads (Addison-Wesley Professional Computing Series)
Python Cookbook
Python in a Nutshell (In a Nutshell (O’Reilly))
Quantum Learning: Unleashing the Genius in You
RESTful Web Services
Rage
Reflections on a Theory of Organisms
Rick Sammon’s Exploring the Light: Making the Very Best In-Camera Exposures
SQL Server Forensic Analysis
Secure Programming with Static Analysis (Addison-Wesley Software Security Series)
Skin: The Complete Guide to Digitally Lighting, Photographing, and Retouching Faces and Bodies
Software Estimation: Demystifying the Black Art (Best Practices (Microsoft))
Standard C++ IOStreams and Locales: Advanced Programmer’s Guide and Reference
Strategy: An Introduction to Game Theory, 2nd Edition
Subject To Change: Creating Great Products & Services for an Uncertain World: Adaptive Path on Design (Adaptive Path)
Surely You’re Joking, Mr. Feynman! (Adventures of a Curious Character)
Surreal Numbers
Swords & Circuitry: A Designer’s Guide to Computer Role-Playing Games (Game Development)
System Center Operations Manager 2007 Unleashed
Teach Yourself Ole Programming in 21 Days/Book
The Art of Debugging with GDB, DDD, and Eclipse
The BIOS Companion: The book that doesn’t come with your motherboard!
The Basque Language: A Practical Introduction (The Basque Series)
The Best of 2600: A Hacker Odyssey
The BetterPhoto Guide to Photographing Children (BetterPhoto Series)
The Burning Shore
The C++ Standard Library Extensions: A Tutorial and Reference
The Game Localization Handbook (Game Development Series)
The God Delusion
The Great Terror: A Reassessment
The Haskell Road to Logic, Maths and Programming (Texts in Computing S.)
The Haskell School of Expression: Learning Functional Programming through Multimedia
The New Turing Omnibus: Sixty-Six Excursions in Computer Science
The Notebooks of Henry James
The Princeton Companion to Mathematics
The Ultimate Guide to Video Game Writing and Design
Three Dragon Ante (Dungeon & Dragons)
Time Management for System Administrators
Traditional Wooden Toys: Their History and How to Make Them
Traffic: Why We Drive the Way We Do (and What It Says About Us)
Types and Programming Languages
WPF in Action with Visual Studio 2008
Why Programs Fail, Second Edition: A Guide to Systematic Debugging
Why Programs Fail: A Guide to Systematic Debugging
Windows  via C/C++ (PRO-Developer) (Pro - Developer)
Windows Forensic Analysis Including DVD Toolkit
Windows Home Server: Protect and Simplify your Digital Life
Windows NT File System Internals (OSR Classic Reprints)
Windows PowerShell in Action
Windows XP Hacks
Witches Abroad
Write Faster, Write Better
Yoga Spandakarika: The Sacred Texts at the Origins of Tantra
bash Cookbook: Solutions and Examples for bash Users (Cookbooks (O’Reilly))
The Failure Factory: How Unelected Bureaucrats, Liberal Democrats, and Big Government Republicans Are Undermining America’s Security and Leading Us to War

- Dmitry Vostokov @ DumpAnalysis.org -

Pictures from Memory Space (Part 2)

Friday, May 29th, 2009

Now some pictures from Citrix CDF traces.

Deep waters of The Sea of Traces:

Considering software tracing as narrative no wonder one day I discovered the vast Library of Software Logs in the sea above:

- Dmitry Vostokov @ DumpAnalysis.org -

Exception Addresses from Event Logs

Thursday, May 28th, 2009

One of the questions that is often asked is about exception addresses in application and system event logs. For example, we have this typical log entry:

"The instruction at "0x77ca8fa7" referenced memory address "0x00000000". The memory could not be read."

Suppose the dump is not available. Can we find the function address to look in our problem database?

My answer here is usually the following:

Even if the application is no longer running we can either noninvasively attach a debugger to it or get a user dump of it and later find the nearest address using ln WinDbg command (remember to apply correct symbols first, see windbg.org):

0:000> ln 77ca8fa7
(77ca8f91)   msvcrt!wcscpy+0×16   |  (77ca8fd6)   msvcrt!wcspbrk

Usually applications crash inside functions and not at their entry addresses, so we pay attention to wcscpy function because it has the offset +0×16.

Note: on Vista and W2K8 due to ASLR, system DLLs could be at different addresses if we take the dump of or attach to a different running process instance.

- Dmitry Vostokov @ DumpAnalysis.org -

The Meaning of DATA

Tuesday, May 26th, 2009

I was suddenly enlightened by the unification of software traces with memory dumps and it came to me that DATA is simply Dump Analysis + Trace Analysis. It is commutative with TADA, the sound of accomplishment (tada.wav in Windows \ Media folder).

This can’t be a coincidence, can it?

- Dmitry Vostokov @ DumpAnalysis.org -

Software Trace - A Mathematical Definition

Tuesday, May 26th, 2009

What is a software trace from a mathematical standpoint? Before any software writes its trace data, it assembles it in memory. Therefore, generally, a software trace is a linear ordered sequence of specifically prepared memory fragments (trace statements):

(ts1, ts2, …, tsn

where every tsi is a sequence of bits, bytes or other discrete units (see the definition of a memory dump):

(s11, s12, …, s1k, s21, s22, …, s2l, …, …, …, sn1, sn2, …, snm)

These trace statements can also be minidumps, selected regions of memory space. In the limit, if every tsi is a full memory snapshot saved at an instant of time (ti) we have a sequence of memory dumps:

(mt1, mt2, …, mtn

Like with memory dump analysis we need symbol files to interpret saved memory fragments unless they were already interpreted during their construction. For example, traces written according ETW specification (Event Tracing for Windows), need TMF files (Trace Message Format) for their interpretation and viewing. Usually these files are generated from PDB files and therefore we have this correspondence:

memory dump file -> software trace file

PDB file -> TMF file 

- Dmitry Vostokov @ TraceAnalysis.org -

Memory Field Theories of Memuonics (Part 1)

Tuesday, May 26th, 2009

Do you remember memuons1, the indivisible entities of memory? Their study is the domain of the new science called memuonics2. According to the so called memophysical principle3,we have particle interpretation of memuons. This is called classical memuonics with classical memory field theory where memuons are “quanta” of memory. We can also ”quantize” memory fields and get quantum memory field theories where memuons are created and annihilated.

(1) The notion of memuons first appeared in the philosophy of memoidealism.

(2) Please don’t confuse memuonics with memiotics. The latter is computer memory semiotics.

(3) Memophysical principle - theories of memory-based universe need to take into account the current mainstream sciences including physics.

- Dmitry Vostokov @ DumpAnalysis.org -

Bugtation No.95

Monday, May 25th, 2009

“A trace is a narrative, the story of a computation.”

Dmitry Vostokov, Software Tracing and Logging: Architecture, Design, Implementation and Analysis Patterns

- Dmitry Vostokov @ TraceAnalysis.org -

Comprehensive Rootkit Book

Monday, May 25th, 2009

Found today this book while browsing Amazon:

The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System

Buy from Amazon

Intrigued, I searched for its table of contents and found the author’s site:

Book TOC

Looks enough comprehensive so I pre-ordered the book and plan to write a review later from windows internals and memory dump analysis perspective.

- Dmitry Vostokov @ DumpAnalysis.org -

Collapsed Stack Trace

Saturday, May 23rd, 2009

This is a stack trace (backtrace) where all finctions are removed and only modules are left. Useful for depicting component dependencies. Here is an example:

0: kd> kc 100
nt!KiSwapContext
nt!KiSwapThread
nt!KeDelayExecutionThread
3rdPartyAVDriver
3rdPartyAVDriver
3rdPartyAVDriver
3rdPartyAVDriver
nt!IofCallDriver
DriverA!Dispatch
DriverA!KUI_dispatch
nt!IofCallDriver
DriverB!PassThrough
DriverB!Dispatch
nt!IofCallDriver
DriverC!LowerPassThrough
DriverC
DriverC
DriverC
DriverC!DispatchPassThrough
nt!IofCallDriver
nt!IopParseDevice
nt!ObpLookupObjectName
nt!ObOpenObjectByName
nt!IopCreateFile
nt!IoCreateFile
nt!NtOpenFile
nt!KiFastCallEntry
ntdll!KiFastSystemCallRet
ntdll!NtOpenFile
ntdll!LdrpCreateDllSection
ntdll!LdrpMapDll
ntdll!LdrpLoadDll
ntdll!LdrLoadDll
kernel32!LoadLibraryExW
kernel32!LoadLibraryW
ntdll!LdrpCallInitRoutine
ntdll!LdrpRunInitializeRoutines
ntdll!LdrpLoadDll
ntdll!LdrLoadDll
kernel32!LoadLibraryExW
kernel32!LoadLibraryW
USER32!LoadAppDlls
USER32!ClientThreadSetup
USER32!__ClientThreadSetup
ntdll!KiUserCallbackDispatcher
nt!KiCallUserMode
nt!KeUserModeCallback
win32k!xxxClientThreadSetup
win32k!xxxCreateThreadInfo
win32k!UserThreadCallout
win32k!W32pThreadCallout
nt!PsConvertToGuiThread
nt!KiBBTUnexpectedRange

Collapsed stack trace is much simpler to grasp: 

nt
3rdPartyAVDriver
nt
DriverA
nt
DriverB
nt
DriverC
nt
ntdll
kernel32
ntdll
kernel32
USER32
ntdll
nt
win32k
nt

- Dmitry Vostokov @ DumpAnalysis.org -

Graphical Notation for Memory Dumps (Part 1)

Saturday, May 23rd, 2009

Inspired by Penrose tensor notation encountered in The Road to Reality book and Feynman diagrams I’d like to introduce Visual Dump Objects (VDO) graphical notation to depict and communicate memory dump analysis patterns, their combinations and analysis results. Let’s look at some basic visual objects (shown in handwriting).

1. A thread:

   or   

2. A function:

3. A module:

4. A thread running through functions, modules or both (stack trace). Optional arrowhead can indicate stack trace direction:

  or    or  

Threads running through modules depict collapsed stack traces.

5. A blocked thread:

An example of 3 threads blocked by another thread (an arrowhead can disambiguate the direction of the waiting chain):

6. A spiking thread (colors are encouraged in VDO notation):

   or   

7. Space boundary between user land and kernel land:

 

Here is an example of the thread spiking in kernel space:

or with modules from stack trace:

More notation to come very soon.

- Dmitry Vostokov @ DumpAnalysis.org -

On Debugging

Saturday, May 23rd, 2009

Instead of publishing another philosophical treatise ”On …”, OpenTask, iterative and incremental publisher, plans to release my collection of bugtations in somewhat extended version by the end of this summer:  

On Debugging: Bugtations and Other Humorous Quotations (ISBN: 978-1906717285)

The book also includes short biographical notes, commentaries and relevant explanations. Hope you would enjoy it. 

- Dmitry Vostokov @ DumpAnalysis.org -

Bugtation No.94

Saturday, May 23rd, 2009

Universal memory dumps come from astronomical observations:

Dumps “are meaningless without” symbols “to interpret them.”

Raymond Arthur Lyttleton, Quoted in Astroparticle Physics, page 50

- Dmitry Vostokov @ DumpAnalysis.org -

Bugtation No.93

Friday, May 22nd, 2009

“Oh, he is a good” troubleshooter, “but he has no power of” debugging!”

Charles Darwin, The life and letters of Charles Darwin, Volume 1

- Dmitry Vostokov @ DumpAnalysis.org -

Comparative Memory Dump Analysis: CPU Spikes

Friday, May 22nd, 2009

Comparative analysis is based on saving several consecutive memory dump files to see similarities and differences. Most often this technique is used for memory leaks, for example, process heap leaks. Here we see another example related to CPU spikes.

One application was spiking CPU and its user dump file was saved manually. !runaway WinDbg command showed the following distribution of kernel and user thread times:

0:000> !runaway f
 User Mode Time
  Thread       Time
  14:f48       0 days 1:22:34.343
  15:f4c       0 days 0:00:45.671
  17:f54       0 days 0:00:41.359
  18:f58       0 days 0:00:33.468
  16:f50       0 days 0:00:24.171
  12:f40       0 days 0:00:17.531
  13:f44       0 days 0:00:07.109
   0:f10       0 days 0:00:00.500
[…]
 Kernel Mode Time
  Thread       Time
  15:f4c       0 days 0:00:51.046
  17:f54       0 days 0:00:41.125
  18:f58       0 days 0:00:32.765
  16:f50       0 days 0:00:23.500
  12:f40       0 days 0:00:18.671
  13:f44       0 days 0:00:07.453
  14:f48       0 days 0:00:02.031
   1:f14       0 days 0:00:00.343
[…]
 Elapsed Time
  Thread       Time
   1:f14       0 days 12:01:42.656
   0:f10       0 days 12:01:42.656
  24:f70       0 days 12:01:42.093
  23:f6c       0 days 12:01:42.093
  22:f68       0 days 12:01:42.093
  21:f64       0 days 12:01:42.093
  20:f60       0 days 12:01:42.093
  19:f5c       0 days 12:01:42.093
  18:f58       0 days 12:01:42.093
  17:f54       0 days 12:01:42.093
  16:f50       0 days 12:01:42.093
  15:f4c       0 days 12:01:42.093
  14:f48       0 days 12:01:42.093
  13:f44       0 days 12:01:42.093
[…]

We see that thread #14 spent 1 hour and 22 minutes in user space and only 2 seconds in kernel space. Its thread stack showed DllA and DllB modules:

0:000> ~14s
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for DllA.dll -
eax=00010da2 ebx=09227bdc ecx=09236bcc edx=0000003a esi=006769e4 edi=00000335
eip=6120b231 esp=00fefd04 ebp=00fefd2c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
DllA!foo+0×3549:
6120b231 8b45ec          mov     eax,dword ptr [ebp-14h] ss:0023:00fefd18=09251a30

0:000> kv
ChildEBP RetAddr  Args to Child             
WARNING: Stack unwind information not available. Following frames may be wrong.
00fefd2c 6120be31 00000063 8633cdb9 006769e4 DllA!foo+0×3549
00fefd64 6120c6db 00fefe00 00006f07 0353f594 DllA!foo+0×4149
00fefdbc 6120d517 00fefe00 00fefde8 8633cd25 DllA!foo+0×49f3
00fefdf8 6120db4d 0353f594 09251848 00000000 DllA!foo+0×582f
00fefe14 6120f280 006769a0 00000000 0000020c DllA!foo+0×5e65
00fefe84 00933436 0000020c 0353f5ac 00000001 DllA!foo+0×7598

00fefec0 008414f1 0353f580 034db5d0 00000003 DllB!read+0xc6
[…]
00feffb8 77e64829 0078f290 00000000 00000000 msvcr71!_endthreadex+0xa0
00feffec 00000000 7c3494f6 0078f290 00000000 kernel32!BaseThreadStart+0×34

These modules belong to different vendors and the code could loop in DllB or in one of DllA functions. In order to show statistically which module is more suspect than the other, we requested 3 consecutive memory dumps of the same spiking process. They revealed the similar uniform pattern:

0:000> !runaway f
 User Mode Time
  Thread       Time
  16:e5c       0 days 1:37:24.640
   0:e1c       0 days 0:00:00.640
  17:e60       0 days 0:00:00.453
  18:e64       0 days 0:00:00.328
  15:e58       0 days 0:00:00.265
  14:e54       0 days 0:00:00.203
[…]
 Kernel Mode Time
  Thread       Time
  16:e5c       0 days 0:00:00.687
  15:e58       0 days 0:00:00.421
  17:e60       0 days 0:00:00.375
  14:e54       0 days 0:00:00.265
  18:e64       0 days 0:00:00.250
   0:e1c       0 days 0:00:00.046
[…]
 Elapsed Time
  Thread       Time
   1:e20       0 days 4:41:11.562
   0:e1c       0 days 4:41:11.562
  18:e64       0 days 4:41:10.875
  17:e60       0 days 4:41:10.875
  16:e5c       0 days 4:41:10.875
  15:e58       0 days 4:41:10.875
  14:e54       0 days 4:41:10.875
  13:e50       0 days 4:41:10.875
  12:e4c       0 days 4:41:10.875
  11:e48       0 days 4:41:10.875
  10:e44       0 days 4:41:10.875
   9:e40       0 days 4:41:10.875
   8:e3c       0 days 4:41:10.875
   7:e38       0 days 4:41:10.875
   6:e34       0 days 4:41:10.875
   5:e30       0 days 4:41:10.875
   4:e2c       0 days 4:41:10.875
   3:e28       0 days 4:41:10.875
   2:e24       0 days 4:41:10.875
[…]

1st dump file: Process Uptime: 0 days 4:41:12.000

0:000> ~16r
eax=07abbb6c ebx=07ab45d4 ecx=07ab4604 edx=00679344 esi=006769e4 edi=00000035
eip=6120b23e esp=0106fd30 ebp=0106fd58 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
DllA!foo+0×3556:
6120b23e 8bcb            mov     ecx,ebx

0:000> ~16kv
ChildEBP RetAddr  Args to Child             
WARNING: Stack unwind information not available. Following frames may be wrong.
0106fd58 6120be31 0000000b 47d6689c 006769e4 DllA!foo+0×3556
0106fd90 6120c6db 0106fe2c 00007329 0342d14c DllA!foo+0×4149
0106fde8 6120d517 0106fe2c 0106fe14 47d66b28 DllA!foo+0×49f3
0106fe24 6120db4d 0342d14c 07abb960 00000000 DllA!foo+0×582f
0106fe40 6120f280 006769a0 00000000 000001d8 DllA!foo+0×5e65
0106feb0 00933436 000001d8 0342d164 00000001 DllA!foo+0×7598

0106feec 008414f1 0342d138 00000000 0340b290 DllB!read+0xc6
[…]
0106ffb8 77e64829 00784be0 00000000 00000000 msvcr71!_endthreadex+0xa0
0106ffec 00000000 7c3494f6 00784be0 00000000 kernel32!BaseThreadStart+0×34

2nd dump file: Process Uptime: 0 days 4:41:42.000

0:000> ~16r
eax=0002550b ebx=00679344 ecx=07ab45d4 edx=00000009 esi=006769e4 edi=00000035
eip=6120b231 esp=0106fd30 ebp=0106fd58 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for DllA.dll -
DllA!foo+0×3549:
6120b231 8b45ec          mov     eax,dword ptr [ebp-14h] ss:0023:0106fd44=07abbb48

0:000> ~16kv
ChildEBP RetAddr  Args to Child             
WARNING: Stack unwind information not available. Following frames may be wrong.
0106fd58 6120be31 0000000b 47d6689c 006769e4 DllA!foo+0×3549
0106fd90 6120c6db 0106fe2c 00007329 0342d14c DllA!foo+0×4149
0106fde8 6120d517 0106fe2c 0106fe14 47d66b28 DllA!foo+0×49f3
0106fe24 6120db4d 0342d14c 07abb960 00000000 DllA!foo+0×582f
0106fe40 6120f280 006769a0 00000000 000001d8 DllA!foo+0×5e65
0106feb0 00933436 000001d8 0342d164 00000001 DllA!foo+0×7598

0106feec 008414f1 0342d138 00000000 0340b290 DllB!read+0xc6
[…]
0106ffb8 77e64829 00784be0 00000000 00000000 msvcr71!_endthreadex+0xa0
0106ffec 00000000 7c3494f6 00784be0 00000000 kernel32!BaseThreadStart+0×34

3rd dump file: Process Uptime: 0 days 4:41:59.000

0:000> ~16r
eax=07abbb6c ebx=07ab4604 ecx=07ab45d4 edx=07ab4604 esi=006769e4 edi=00000035
eip=6120b239 esp=0106fd30 ebp=0106fd58 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
DllA!foo+0×3551:
6120b239 895108          mov     dword ptr [ecx+8],edx ds:0023:07ab45dc=07ab4604

0:000> ~16kv
ChildEBP RetAddr  Args to Child             
WARNING: Stack unwind information not available. Following frames may be wrong.
0106fd58 6120be31 0000000b 47d6689c 006769e4 DllA!foo+0×3551
0106fd90 6120c6db 0106fe2c 00007329 0342d14c DllA!foo+0×4149
0106fde8 6120d517 0106fe2c 0106fe14 47d66b28 DllA!foo+0×49f3
0106fe24 6120db4d 0342d14c 07abb960 00000000 DllA!foo+0×582f
0106fe40 6120f280 006769a0 00000000 000001d8 DllA!foo+0×5e65
0106feb0 00933436 000001d8 0342d164 00000001 DllA!foo+0×7598

0106feec 008414f1 0342d138 00000000 0340b290 DllB!read+0xc6
[…]
0106ffb8 77e64829 00784be0 00000000 00000000 msvcr71!_endthreadex+0xa0
0106ffec 00000000 7c3494f6 00784be0 00000000 kernel32!BaseThreadStart+0×34

We see exactly the same stack trace except the first (top) frame. The offset differs very little: +0×3556, +0×3549 (twice) and +0×3551. We can suspect the small loop here:

0:000> ub
DllA!foo+0x3549:
6120b231 8b45ec          mov     eax,dword ptr [ebp-14h]
6120b234 8d0490          lea     eax,[eax+edx*4]
6120b237 8b10            mov     edx,dword ptr [eax]
6120b239 895108          mov     dword ptr [ecx+8],edx
6120b23c 8908            mov     dword ptr [eax],ecx
6120b23e 8bcb            mov     ecx,ebx
6120b240 85db            test    ebx,ebx
6120b242 75e3            jne     DllA!foo+0x353f (6120b227)

0:000> ub 6120b231
DllA!foo+0x3535:
6120b21d 8b4d08          mov     ecx,dword ptr [ebp+8]
6120b220 8b0c88          mov     ecx,dword ptr [eax+ecx*4]
6120b223 3bcb            cmp     ecx,ebx
6120b225 741d            je      DllA!foo+0x355c (6120b244)
6120b227 8b5908          mov     ebx,dword ptr [ecx+8]
6120b22a 8b410c          mov     eax,dword ptr [ecx+0Ch]
6120b22d 33d2            xor     edx,edx
6120b22f f7f7            div     eax,edi

0:000> u
DllA!foo+0x3549:
6120b231 8b45ec          mov     eax,dword ptr [ebp-14h]
6120b234 8d0490          lea     eax,[eax+edx*4]
6120b237 8b10            mov     edx,dword ptr [eax]
6120b239 895108          mov     dword ptr [ecx+8],edx
6120b23c 8908            mov     dword ptr [eax],ecx
6120b23e 8bcb            mov     ecx,ebx
6120b240 85db            test    ebx,ebx
6120b242 75e3            jne     DllA!foo+0x353f (6120b227)

0:000> u 6120b242
DllA!foo+0x355a:
6120b242 75e3            jne     DllA!foo+0x353f (6120b227)
6120b244 ff4508          inc     dword ptr [ebp+8]
6120b247 8b4508          mov     eax,dword ptr [ebp+8]
6120b24a 3b4608          cmp     eax,dword ptr [esi+8]
6120b24d 72cc            jb      DllA!foo+0x3533 (6120b21b)
6120b24f ff36            push    dword ptr [esi]
6120b251 e88dc70000      call    DllA!foo+0xfcfb (612179e3)
6120b256 59              pop     ecx

All this strongly rules out DllB and points to DllA as the first vendor contact.

- Dmitry Vostokov @ DumpAnalysis.org -

e-acheri

Friday, May 22nd, 2009

This is a computational ghost that comes from CPU spiking tops and hills to bring sickness to small programs. Red color from RGB triplets provides early warning signs.

Inspired by non-computational Acheri.

PS. This category of computational ghosts and bug hauntings was also inspired by an e-mail conversation between two software engineers that I witnessed some years ago when one told another: “Is this your fix that is still haunting us?”. If you have ever visited heated code review debates you can imagine the provoked response.

- Dmitry Vostokov @ DumpAnalysis.org -

Pictures from Memory Space (Part 1)

Tuesday, May 19th, 2009

Here are some artistic images found in process memory dumps of VLC media player and IE. The latter was freezing after memory leaking, the size of its memory dump file is more than 1.5Gb.

Needle (VLC)

The tip of the same Needle magnified by x8:

The Image of Heap (from IE):

Red Zones (from IE):

- Dmitry Vostokov @ DumpAnalysis.org -

Software Tracing and Logging

Monday, May 18th, 2009

This is a forthcoming book to be released next year after we finally publish DebugWare book by the end of this summer:

Software Tracing and Logging: Architecture, Design, Implementation and Analysis Patterns (ISBN: 978-1906717728)

I have already begun working on it in the background. The scope of DebugWare book is too wide to cover tracing and logging in great detail not to mention the very important subject of software trace analysis.

- Dmitry Vostokov @ TraceAnalysis.org -

Patterns of Computer Memory

Monday, May 18th, 2009

Last weekend I came across this book in a local bookshop:

Patterns Of The Earth

Buy from Amazon

I immediately recalled all beautiful patterns I have seen and collected from visualized computer memory maps. I plan to include many images in a full-color book this summer:

Patterns of Computer Memory (ISBN: 978-1906717711)

This book should be essential for creative computer engineers and inquiring debugging minds!

- Dmitry Vostokov @ DumpAnalysis.org -

1000 Blog Posts: T&D Labyrinth

Monday, May 18th, 2009

This post number is 1,000. Wordpress admin panel reports that I have written 999 (excluding this one):

Of course, I wrote my 1,000th blog post some time ago if I count all my 7 blogs but this is 1,000th post for just this blog.

To celebrate this event, I’m posting a picture of troubleshooting and debugging labyrinth resting on a notion of universal memory dumps that are observational snapshots and include both memory and various traces we collect to resolve problems.

 

This picture shows how we arrive to problem resolution. For example:  

I’m also working on another picture, called T&D Tangram :-)

- Dmitry Vostokov @ DumpAnalysis.org -