Software Diagnostics Library

I’m pleased to announce that MDAA, Volume 4 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4

It features:

- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art

Its table of contents is available here:

http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I detour for MARS expedition. You may also call it Memory Analysis Report System as malware analysis is always exploration of memory (in general). Why is this sudden change of course? After reading Gilles Deleuze I want to broaden the concept of “malware” and give it new orientation and direction of thinking. Beside that I also want new challenges after many years of research in pattern-driven memory dump and software trace analysis of abnormal software behaviour.

You may have also noticed small restructuring (rebranding) of this blog and DumpAnalysis.org headers.

See you there

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A Momentary Lapse of Computation.

Pink Floyd, A Momentary Lapse of Reason

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally, after the delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Multithreading

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There are many different approaches to illustrate virtual to physical memory mapping on systems with paging like Windows. Here is another approach that uses natural memory visualization. An image of a user process was generated and juxtaposed to an image of kernel memory dump generated afterwards to produce the combined picture of the full virtual space. Of course, uncommited regions were not included in it as they were not present in user and kernel dumps. Then, after reboot, the same application was launched again and an image of a complete memory dump was generated. Finally, both images were juxtaposed to produce this approximate picture:

In the virtual memory space to the left we see much more granularity. On the contrary, the physical memory space to the right is more uniform and has a different coloring.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

My drive to generalization led me to place an adornment on the portal to highlight the fact that memory and software trace analysis patterns are under an umbrella of general software behaviour patterns:

http://www.dumpanalysis.org/Software-Behavior-Patterns-Headline

In the forthcoming post series I plan to write about similarities between these two branches and also provide pattern examples from non-Windows platforms. All this material will provide the foundation for the forthcoming book Software Behavior: A Guide to Systematic Analysis (ISBN: 978-1906717162).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Yesterday I discovered the blog j00ru//vx where I was pleased to see another memory visualization approach which I classify as synthetic:

x86 Kernel Memory Space Visualization (KernelMAP v0.0.1)

So far now I put a bit more extended (but in no way complete) classification with links (based on my previous blog post where every category is presented in chronological order of my encounter with links):

1. Synthetic

• Numerous debuggers that show memory maps for easy navigation (links to follow)
• VMMap
• KernelMAP v0.0.1
• MemSpyy
• Virtual memory usage and GC heap usage

2. Natural

a. Static

• Dump2Picture (Windows)
• dump2pic (Linux)
• 2D and 3D visualization using general-purpose tools like ParaView.

b. Semi-dynamic

• WinDbg scripts

c. Dynamic 

• Haywire

Please let me know any other approaches or links you know. 

PS. I’m currently a big fan of artificial evolution and recommend this fantastic full-color book that has good ideas about expression-based visualization:

The Art of Artificial Evolution: A Handbook on Evolutionary Art and Music (Natural Computing Series)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming CARE and STARE online systems additionally aim to provide software behaviour pattern identification via debugger log and trace analysis and suggest possible software troubleshooting patterns. The purpose of these post series is to provide high level overview of possible patterns of software behavior and how they can be recognised and analyzed. This work started in October, 2006 with the identification of computer memory patterns and later continued with software trace patterns. Bringing all of them under a unified linked framework seems quite natural to me.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

• Acorn (The Korean translation publisher)
• Kyobo book
• Yes24.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I originally intended to name this blog post as ”What I’m Reading Now” but then decided to show it as another satisfying example of my Mod N Reading technique. During my 7 years in memory dump analysis captivity I didn’t pay much attention to traditional synthetic software engineering (as opposed to analytical software defect research in computer memory) except occasionally writing some troubleshooting tools, describing DebugWare patterns in UML and devising RADII process. A few weeks ago I decided to brush up my engineering skills and read some books that accumulated in my library during last few years. Here is the list of them (debugging triptych of Windows Internals 5th Edition, Advanced Windows Debugging, and Advanced .NET Debugging are on my office table and I read them almost daily so I’m not including them in the list below).

Illustrated Mod N is actually Mod 7 technique where I cycle through 7 topics with 3 books for each topic. Ideally I aim to dedicate one topic per day every week but this is not always possible due to writing and publishing but I still do it in a Mod 7 way even if I skip some days. it usually takes me an hour or two to read carefully 5-10 pages from each of 3 topical books. Here is the current state of the reading round-robin queue (21 books) under my home computer desk:

Here are the topics and corresponding books (with links if you would like to buy them from Amazon):

Multithreading from Computer Science Perspective

Synchronization Algorithms and Concurrent Programming

Modern Multithreading : Implementing, Testing, and Debugging Multithreaded Java and C++/Pthreads/Win32 Programs

The Art of Multiprocessor Programming

Algorithms, Parsing

Algorithms in a Nutshell

Flex & Bison: Text Processing Tools

The Algorithm Design Manual

Statistics 

Statistics in a Nutshell: A Desktop Quick Reference

Statistics Hacks: Tips & Tools for Measuring the World and Beating the Odds

Statistics, 4th Edition

C++, STL and Boost 

C++ in a Nutshell

Beyond the C++ Standard Library: An Introduction to Boost

C++ Cookbook

Security, Mac OS X

The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System

The Mac Hacker’s Handbook

Security Engineering: A Guide to Building Dependable Distributed Systems

Code, Games

Programming Language Pragmatics, Third Edition

Game Engine Architecture

Code Complete: A Practical Handbook of Software Construction

Embedded and Real-Time Software Engineering

Designing Embedded Hardware

Bebop to the Boolean Boogie, Third Edition: An Unconventional Guide to Electronics

Software Engineering for Real-Time Systems

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

• Title: Memory Dump Analysis Anthology, Volume 4
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 410 pages
• Publisher: Opentask (30 March 2010)
• ISBN-13: 978-1-906717-86-5

• Hardcover: 410 pages
• Publisher: Opentask (30 April 2010)
• ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having considered computational threads as braided strings and after discerning several software trace analysis patterns (just the beginning) we can see formatted and tabulated software trace output in a new light and employ the “fabric of traces” and braid metaphors for an Adjoint Thread concept. This new concept was motivated by reading about Extended Phenotype (*) and extensive analysis of Citrix ETW-based CDF traces using CDFAnalyzer. The term Adjoint was borrowed from mathematics because the concept we discuss below resembles this metaphorical formula: (Thread A, B) = [A, Thread B]. Let me first illustrate adjoint threading using simplified trace tables. Consider this generalized software trace example (date and time column is omitted for visual clarity):

We see several threads in a process PID 2792. In CDFAnalyzer we can filter trace messages that belong to any column and if we filter by TID we get a view of any Thread of Activity. However, each thread can “run” through any source directory, file name or function. If a function belongs to a library multiple threads would access it. This source location (can be considered as a subsystem), file or function view of activity is called an Adjoint Thread. For example, if we filter only subsystemA column in the trace above we get this table:

We can graphically view subsystemA as a braid string that “permeates the fabric of threads”:

We can get many different braids by changing filters, hence multibraiding. Here is another example of a driver source file view initially permeating 2 process contexts and 4 threads:

(*) A bit of digression. Looks like biology keeps giving insights into software, there is even a software phenotype metaphor albeit a bit restricted to code, I just thought that we need also an Extended Software Phenotype.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Main topics of Debugged! MZ/PE magazine issues for June and September 2010:

Debugged! MZ/PE: Multiplatform Software Defects, June, 2010 (Paperback, ISBN: 978-1906717902)

Debugged! MZ/PE: Software Defect Visualization and Annotation, September, 2010 (Paperback, ISBN: 978-1906717919)

Here are draft front covers designed today:

- Dmitry Vostokov @ DumpAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday.

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

Finally, after the long delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Software Tracing

- Dmitry Vostokov @ DumpAnalysis.org -

The CRC Encyclopedia of Mathematics, Third Edition - 3 Volume Set

I spotted this book on Amazon US and was thrilled to have the new edition in 3 volumes for easy handling when reading. I also have the previous edition that is even featured on my own book cover (the picture of my previous library book arrangement, the book is highlighted in white rectangle in the lower right corner):

This is a unique volume that sits between The Princeton Companion to Mathematics (that I’m also reading now) and Encyclopedic Dictionary of Mathematics: The Mathematical Society of Japan (that I also own). In fact after reading 3 volumes from cover to cover I can start with 2 volumes of EDM. There is also Springer Encyclopaedia of Mathematics in 11 volumes with various additional supplement volumes that I plan to own as well and it looks to me on the same level as EDM.

After searching for the best price I ordered a copy from Amazon DE and after my purchase in just a few days the price was up by 50%! I can only explain this that more people tried to purchase after I used twitter to announce this encyclopedia (there were 5 copies available on Amazon DE and in just 2 days only 1 left) or there was a mistake in price.

3 volumes arrived and I immediately started reading them, a few pages from each volume every day using mod 3 reading technique, for example, Wed - Vol I, Thu - Vol II, Fri - Vol III, Mon - Vol I, an so on. I prefer paper books for bulk reading instead of electronic version (in this case corresponding website) although if I’m interested in a specific article or a keyword I go to Wolfram MathWorld website to get the latest update and citations. These paperback volumes are just for day-to-day scheduled reading to get ideas and general mathematical education. This is why I don’t need an Index. For example, just after reading the first pages I got the idea of cubic (qubic) memory representation.

I usually put reviews on Amazon after I finish a book from cover to cover but in this case the review would be waiting for at least a year so I write it now based on my first impressions. After some time I plan to update it.

- Dmitry Vostokov @ LiterateScientist.com -

A nibble is a (0,1)-matrix, a byte is a cubic 0,1-lattice and the next cubic byte-boundary 0,1-lattice represents a 64-bit qword:

This is what I call a natural memory representation as memory building blocks or qubic memory (do not mistaken it with qubit memory). This elevates bytes and 64-bit quadruple words as natural addresses and shows that 32-bit addresses are unnatural.

This also allows to us to visualize certain overlapped memory patterns in dump files (same vertice, edge or side).

- Dmitry Vostokov @ DumpAnalysis.org -

To commemorate 200 years since Darwin’s birth and 150 years since the publication of The Origin of Species, OpenTask plans to dedicate December, 2009 issue of Debugged! MZ/PE magazine to taxonomy and evolution of software defects and March, 2010 issue to multithreading (as natural selection).

Debugged! MZ/PE: Systematics and Evolution of Software Defects, December, 2009 (Paperback, ISBN: 978-1906717827)

Debugged! MZ/PE: Multithreading, March, 2010 (Paperback, ISBN: 978-1906717834)

Here are draft front covers designed during last weekend:

- Dmitry Vostokov @ DumpAnalysis.org -