Projective Debugging

Modern software systems and products are hard to debug despite their elaborate tracing and logging facilities. Typical logs may include millions of trace messages from hundreds and thousands of components, processes, and threads. The postmortem diagnostic analysis became more structural after the introduction of Trace and Log Analysis Patterns but live debugging requires a lot more efforts. Here we introduce Projective Debugging as a tool for trace-level debugging. Its main idea is to analyze, diagnose and debug the so-called “projected” execution of software as seen from the original software execution traces and logs:

Picture 1. Original software execution is mapped into projected software as seen from traces and logs.

Please notice, that Projective Debugging is different from the so-called Prototype Debugging by creating models after the software product is built (some engineering methodologies prescribe that prototypes should be discarded before building the product):

Picture 2. Prototype software is mapped into the product.

The problems diagnosed and solved in the projected system are fed back into the original system:

Picture 3. Debugged projected software is mapped into the original software.

The implementation of the main idea of Projective Debugging is that: we can take a trace or log, interpret every trace message according to some rules, and translate it into executable code mirroring components and execution entities such as user sessions, processes, and threads. This is a task for the Projective Debugger, and it is illustrated in the following diagram where we borrowed notation from UML:

Picture 4. The logs and traces from the original product execution are translated by Projective Debugger.

For example, the Projective Debugger (ProjectiveDebugger.exe) interprets these very simple messages below, and creates a process PID220.exe (we have only one thread), and then opens a file “data.txt”. After 10 seconds, it closes the file.

Time     PID  TID  Message 
-----------------------------------
11:00:00 220  330  Open “data.txt”
11:00:10 220  330  Close “data.txt”

In addition to executing code corresponding to messages using the same time deltas as read from the trace, it may scale execution time proportionally, for example, executing 2-day log in a matter of minutes. Such scaling may also uncover additional synchronization bugs.

The trace may be pre-processed, and all necessary objects created before execution or it may be interpreted line by line. For complex traces, the projected source code may be generated for later compilation, linking, and execution. Once the projected code is executed, breakpoints may be set on existing traces, and other types of Debugging Implementation Patterns may be applied. Moreover, we may re-execute the trace several times and even have some limited form of backward execution.

The resulting code model can be inspected by native debuggers for the presence of Memory Analysis Patterns and can even have its own logging instrumentation with traces and logs analyzed by another instance of the Projective Debugger:

Picture 5. Projected Product Execution is inspected by a native debugger and also generates its own set of traces and logs to be projected to another model by another instance of the Projected Debugger.

We created the first version of the Projective Debugger and successfully applied to a small trace involving synchronization primitives across several threads. The Projective Debugger was able to translate it into an executable model with the same number of threads and the same synchronization primitives and logic. The resulting process was hung as expected and we attached a native debugger (WinDbg for Windows) and found a deadlock.

Since traces are analyzed from platform-independent Software Narratology perspective it is possible to get a trace from one operating system, and then, after applying a set of rules, re-interpret it into an executable model on another operating system. We created the similar multithreading test program on Mac OS X that was hung and reinterpreted its trace into an executable model under Windows:

Picture 6. The traces and logs from the original product execution on Mac OS X are projected by a Windows version of Projective Debugger into an executable model for a Windows platform.

Since resultant executable models can also have corresponding logging instrumentation mirroring original tracing, any problems found in executable models can be fixed iteratively and, once the problem disappears, the resulting fix or configuration may be incorporated into the original product or system.

If tracing involves kernel space and mode, a specialized projected executable can be created to model the operating system and driver level.

The more tracing data you have, more real your projected execution becomes. However, we want to have enough tracing statements but not to complicate the projected model. Then, ideally, we should trace only relevant behavior, for example, corresponding to use cases and architecture.

Projective Debugging may also improve your system or product maintainability by highlighting whether you need more tracing statements and whether you need more accurate and complete tracing statements.