Archive for February, 2012

Bugtation No.156

Wednesday, February 22nd, 2012

Philosophy is about memories, people’s memories.

Dmitry Vostokov, The Last Presocratic, Founder of Memodealism

- Dmitry Vostokov @ + -

Commodities as Memories (Part 1)

Tuesday, February 21st, 2012

Whereas Marx considered commodities as having surplus values and Baudrillard considered them as semiotic signs, philosophy of Memoidealism considers them as containing memories and linked by memories. People use commodities to evoke memories (either personal or collective) and use them for further communication. You can find such examples everywhere and the use of memories (including superstructure and means of production memories used to store the personal ones) has increased dramatically. We call this analysis XRAM to distinguish from original Marx analysis and its derivatives and extensions. In the future memories will also have exchange value and replace money.

- Dmitry Vostokov @ + -

Watching a Movie (Debugging Slang, Part 29)

Monday, February 20th, 2012

Watching a Movie - Watching the prodigious output of some debugging commands and scripts in real time.

Examples: Watching the output of !process 0 ff  WinDbg command. Watching the output of user stack trace database and breaking in when it becomes uniform.

- Dmitry Vostokov @ + -

The Design of Memory Dump Analysis: 7 Steps of Highly Successful Analysts

Monday, February 20th, 2012

I was recently asked by a group of trainees to outline a simple approach to proceed after opening a memory dump. So I came up with these 7 steps:

1. !analyze -v [-hang]
2. Exception (Bugcheck): stack trace analysis with d* and lmv
3. !locks
4. !runaway f (!running)
5. Dump all (processes and) thread stack traces [with 32-bit] ~*kv (!process 0 ff)
6. Search for signs/patterns of abnormal behavior (exceptions, wait chains, message boxes [, from your custom checklist])
7. Narrow analysis down to a specific thread and dump raw stack data if needed [repeat*]

(commands/options in brackets denote kernel/complete dump variation)
[notes in square brackets denote additional options, such as x64 specifics, your product details, etc.]

What are your steps? I would be interested to hear about alternative analysis steps, techniques, etc.

- Dmitry Vostokov @ + -

Webinar Introduction to Software Narratology: An Applied Science of Software Stories

Sunday, February 19th, 2012

Memory Dump Analysis Services ( organizes a free Webinar about software narratology: an application of ideas from narratology to software narrative stories.

Introduction to Software Narratology: An Applied Science of Software Stories Logo

Learn about this exciting new discipline and a field of research. When software executes it gives us its stories in the form of UI events, software traces and logs. Such stories can be analyzed for their structure and patterns for troubleshooting, debugging and problem resolution purposes.

Prerequisites: basic software troubleshooting and ability to read software generated logs.

Audience: anyone can benefit from this new perspective: from software users to software engineers and managers.

Space is limited.
Reserve your Webinar seat now at:

- Dmitry Vostokov @ + -

Postmortem effects of -g

Sunday, February 19th, 2012

One of attendees of accelerated memory dump analysis training pointed me to the possible effects of -g option for AeDebug custom postmortem debugger command line for CDB, NTSD or WinDbg. So I tested that with x64 TestWER tool (should be the same with x86 version) and indeed there are differences.

With -g option with have this stack trace:

AeDebug\Debugger = "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -p %ld -e %ld -g

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012f210 00000001`40004148 TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0x7e
00000000`0012f250 00000001`40004388 TestWER64!_AfxDispatchCmdMsg+0xc4
00000000`0012f280 00000001`40003552 TestWER64!CCmdTarget::OnCmdMsg+0x180
00000000`0012f2e0 00000001`4000cc44 TestWER64!CDialog::OnCmdMsg+0x32
00000000`0012f320 00000001`4000d877 TestWER64!CWnd::OnCommand+0xcc
00000000`0012f3b0 00000001`40008c2c TestWER64!CWnd::OnWndMsg+0x5f
00000000`0012f4f0 00000001`4000c272 TestWER64!CWnd::WindowProc+0x38
00000000`0012f530 00000001`4000c32d TestWER64!AfxCallWndProc+0xfe
00000000`0012f5d0 00000000`77519bd1 TestWER64!AfxWndProc+0x59
00000000`0012f610 00000000`77516aa8 USER32!UserCallWinProcCheckWow+0x1ad
00000000`0012f6d0 00000000`77516bad USER32!SendMessageWorker+0x682
00000000`0012f760 000007fe`fccb0bbf USER32!SendMessageW+0x5c
00000000`0012f7b0 000007fe`fccb47df COMCTL32!Button_ReleaseCapture+0x157
00000000`0012f7f0 00000000`77519bd1 COMCTL32!Button_WndProc+0xcbf
00000000`0012f8b0 00000000`775198da USER32!UserCallWinProcCheckWow+0x1ad
00000000`0012f970 00000000`775167c2 USER32!DispatchMessageWorker+0x3b5
00000000`0012f9f0 00000001`400079cc USER32!IsDialogMessageW+0x153
00000000`0012fa80 00000001`40009148 TestWER64!CWnd::IsDialogMessageW+0x38
00000000`0012fab0 00000001`40003513 TestWER64!CWnd::PreTranslateInput+0x28
00000000`0012fae0 00000001`4000b696 TestWER64!CDialog::PreTranslateMessage+0xc3
00000000`0012fb10 00000001`40004c1f TestWER64!CWnd::WalkPreTranslateTree+0x3a
00000000`0012fb40 00000001`40004c7f TestWER64!AfxInternalPreTranslateMessage+0x67
00000000`0012fb70 00000001`40004e26 TestWER64!AfxPreTranslateMessage+0x23
00000000`0012fba0 00000001`40004e6b TestWER64!AfxInternalPumpMessage+0x3a
00000000`0012fbd0 00000001`4000aba6 TestWER64!AfxPumpMessage+0x1b
00000000`0012fc00 00000001`40003e4a TestWER64!CWnd::RunModalLoop+0xea
00000000`0012fc60 00000001`40024da4 TestWER64!CDialog::DoModal+0x1c6
00000000`0012fd10 00000001`40024625 TestWER64!CTestDefaultDebuggerApp::InitInstance+0xc4
00000000`0012fe70 00000001`400153c2 TestWER64!AfxWinMain+0x75
00000000`0012feb0 00000000`77ad652d TestWER64!__tmainCRTStartup+0x186
00000000`0012ff60 00000000`77c0c521 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

0:000> r
rax=0000000000000000 rbx=0000000000000001 rcx=000000000012fd50
rdx=00000000000003e8 rsi=000000000012fd50 rdi=000000014002daa0
rip=00000001400247ae rsp=000000000012f210 rbp=0000000000000111
r8=0000000000000000  r9=0000000140024730 r10=0000000140024730
r11=000000000012f310 r12=0000000000000000 r13=00000000000003e8
r14=0000000000000110 r15=0000000000000001
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244
00000001`400247ae c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=????????

Without -g option we also see exception processing (highlighted in red):

AeDebugger\Debugger = "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -p %ld -e %ld

0:000> kL
Child-SP RetAddr Call Site
00000000`0012e368 000007fe`fe301420 ntdll!ZwWaitForMultipleObjects+0xa
00000000`0012e370 00000000`77ae2cf3 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`0012e470 00000000`77b590f5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`0012e500 00000000`77b59277 kernel32!WerpReportFaultInternal+0×215
00000000`0012e5a0 00000000`77b592cf kernel32!WerpReportFault+0×77
00000000`0012e5d0 00000000`77b594ec kernel32!BasepReportFault+0×1f
00000000`0012e600 00000000`77c743b8 kernel32!UnhandledExceptionFilter+0×1fc
00000000`0012e6e0 00000000`77bf85a8 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`0012e710 00000000`77c09d0d ntdll!_C_specific_handler+0×8c
00000000`0012e780 00000000`77bf91af ntdll!RtlpExecuteHandlerForException+0xd
00000000`0012e7b0 00000000`77c31278 ntdll!RtlDispatchException+0×45a
00000000`0012ee90 00000001`400247ae ntdll!KiUserExceptionDispatcher+0×2e

00000000`0012f450 00000001`40004148 TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0×7e
00000000`0012f490 00000001`40004388 TestWER64!_AfxDispatchCmdMsg+0xc4
00000000`0012f4c0 00000001`40003552 TestWER64!CCmdTarget::OnCmdMsg+0×180
00000000`0012f520 00000001`4000cc44 TestWER64!CDialog::OnCmdMsg+0×32
00000000`0012f560 00000001`4000d877 TestWER64!CWnd::OnCommand+0xcc
00000000`0012f5f0 00000001`40008c2c TestWER64!CWnd::OnWndMsg+0×5f
00000000`0012f730 00000001`4000c272 TestWER64!CWnd::WindowProc+0×38
00000000`0012f770 00000001`4000c32d TestWER64!AfxCallWndProc+0xfe
00000000`0012f810 00000000`77519bd1 TestWER64!AfxWndProc+0×59
00000000`0012f850 00000000`77516aa8 USER32!UserCallWinProcCheckWow+0×1ad
00000000`0012f910 00000000`77516bad USER32!SendMessageWorker+0×682
00000000`0012f9a0 00000000`7751eda7 USER32!SendMessageW+0×5c
00000000`0012f9f0 00000001`400079cc USER32!IsDialogMessageW+0×85f
00000000`0012fa80 00000001`40009148 TestWER64!CWnd::IsDialogMessageW+0×38
00000000`0012fab0 00000001`40003513 TestWER64!CWnd::PreTranslateInput+0×28
00000000`0012fae0 00000001`4000b696 TestWER64!CDialog::PreTranslateMessage+0xc3
00000000`0012fb10 00000001`40004c1f TestWER64!CWnd::WalkPreTranslateTree+0×3a
00000000`0012fb40 00000001`40004c7f TestWER64!AfxInternalPreTranslateMessage+0×67
00000000`0012fb70 00000001`40004e26 TestWER64!AfxPreTranslateMessage+0×23
00000000`0012fba0 00000001`40004e6b TestWER64!AfxInternalPumpMessage+0×3a
00000000`0012fbd0 00000001`4000aba6 TestWER64!AfxPumpMessage+0×1b
00000000`0012fc00 00000001`40003e4a TestWER64!CWnd::RunModalLoop+0xea
00000000`0012fc60 00000001`40024da4 TestWER64!CDialog::DoModal+0×1c6
00000000`0012fd10 00000001`40024625 TestWER64!CTestDefaultDebuggerApp::InitInstance+0xc4
00000000`0012fe70 00000001`400153c2 TestWER64!AfxWinMain+0×75
00000000`0012feb0 00000000`77ad652d TestWER64!__tmainCRTStartup+0×186
00000000`0012ff60 00000000`77c0c521 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

I now prefer omitting -g option to get stack traces equivalent to manual crash dumps saved by userdump.exe on pre-Vista platforms and Task Manager on later platforms.

- Dmitry Vostokov @ + -

Software Anti-Narrative

Friday, February 17th, 2012

In narratology anti-narrative denotes a narrative having sequences of events impossible in reality. In software traces such sequences usually depict abnormal software behaviour. Here are some parallels with corresponding trace analysis patterns:

Fiction                     | Software Trace
Repeated unrepeatable       | Periodic Error (?)
Denarration (erasure)       | No Activity / Incomplete History
Chronological contradiction | Impossible Trace

Question mark means that possibly another pattern is needed there.

- Dmitry Vostokov @ + -

Software Narratology helps Fiction Writers

Monday, February 13th, 2012

Any DA+TA source (dump artifact + trace artifact) can be used to generate and validate narrative fiction where memory dump components provide fiction structure and software trace components provide behaviour (plot and story):

I already use such help in writing fiction books (”computational” novels) and use trace analysis patterns in validating historical narratives (with Excel and software trace viewers as tools). Here fiction is in no way different from software fictions outlined in the article about software stories. Please also note 2 other fields along with software narratology (SN was defined in mid 2009 where the first pattern appeared earlier in 2009) that are under development: software chorology and chorographyAdjoint threads and trace analysis patterns are the major contributions from SN.

Final note: Because we consider memory dumps in a general sense (e.g. philosophy of memoidealism where any data is a memory dump) and software trace narremes as small memory dumps (mid 2009) any data stream potentially can be represented as some fictional or real story. And vice versa, any story can be transformed into corresponding computer memory dumps and software traces (for example, the narralog language is being designed to model software stories).

- Dmitry Vostokov @ + -

Crash Dump Analysis Patterns (Part 167)

Sunday, February 12th, 2012

Regular Data pattern generalizes ASCII and UNICODE-type (00xx00yy) data found in memory to domain-specific data formats such as bitmaps and vector data. An example of the latter could be the sequence of …0xxx0yyy… (xxx are triplets of hex digits). A typical usage of this pattern is analysis of corrupt dynamic memory blocks (process heap, kernel pool) where continuity of regular data across block boundary points to a possible buffer overwrite.

- Dmitry Vostokov @ + -

Crash Dump Analysis Patterns (Part 166)

Sunday, February 12th, 2012

Runtime software exceptions (such as C++ exceptions) can be translated by custom exception handlers into other exceptions by changing exception data. This is different from nested exceptions where another exception is thrown. One example of such possible translation I recently encountered when looking at a raw stack data (!teb -> dps) having signs of hidden exceptions (multiple RaiseException calls) and also CLR execution residue (valid return addresses of clr module). In addition of final invalid handle exception and one hidden access violation there were many exception codes c0000027. Google search pointed to the article about skipped C++ destructors written by S. Senthil Kumar that prompted me to introduce the pattern Translated Exception.

- Dmitry Vostokov @ + -

Narralog - A Software Trace Modeling Language

Thursday, February 2nd, 2012

Forthcoming Accelerated Software Trace Analysis training requires extensive real life like software logs with multiple software behaviour patterns. The similar accelerated memory dump analysis trainings (unmanaged / native Windows and .NET) also required good memory dumps but this problem was solved by modeling patterns of abnormal software behaviour in an appropriate implementation language such as C++ and C#. Modeling software traces with hundreds of software components, processes and threads would require enormous programming efforts. Therefore, the natural approach is to describe a desired software trace in some declarative language (or minimally imperative) and get a million line software log that models a specific combination of trace analysis patterns. So, welcome to such a language called Narralog: Software Narrative Log  or Narrative Logic. Please note that this language is different from Riemann programming language which is a language to describe software problems and generate software problem solving tools. Language details and whether a kind of a Metadefect Template Library will be used to implement Narralog or simple macroprogramming is suffice will be decided and announced when we further develop our training.

- Dmitry Vostokov @ + -

Narremes in Software Narratology

Thursday, February 2nd, 2012

John Randall’s blog about narratology and narremes suggested me to elaborate on a basic software trace (log) unit. One candidate is a trace message invariant, a skeleton trace message similar to a format string. There is also a corresponding software trace analysis pattern called Message Invariant. Although, this might be too an elementary unit akin to a sentence level and another candidate is a macromessage, a combination of several messages serving some semantic function. There is a corresponding general pattern Macrofunction and an example concrete analysis pattern called Exception Stack Trace. The actual software narreme might be situated between these two extremes: invariants and macrofunctions.

If you are new to a software narratology field please look at these posts placed in chronological order (except a pattern catalog):

- Dmitry Vostokov @ + -

Holy Pilgrimage to Harvard University

Wednesday, February 1st, 2012

I have plans for a debugging tour to visit famous debugging places around the world. According to Wikipedia I have to start with Harward University where the first bug was found. I’ll keep you informed on the progress of this memorianic pilgrimage.

- Dmitry Vostokov @ + -

Bugtation No.155

Wednesday, February 1st, 2012

On natural pointers vs. pointers that are not natural, like a NULL, wild or dangling pointer.

Thus, a man who intends keeping pointers naturally tries to get as good debuggers as he can … .

Charles Darwin, On the Origin of Species

- Dmitry Vostokov @ + -

Forthcoming Accelerated Mac OS X Core Dump Analysis training

Wednesday, February 1st, 2012

Memory Dump Analysis Services plans to introduce Mac OS X memory dump analysis training this year based on the success of its Windows variant. For details and how to register please visit this page:

- Dmitry Vostokov @ + -