Software Diagnostics Library

Sometimes the assembly code looks almost wild (not like generated by your favourite compiler). For example (that also shows .NET runtime native unhandled exception processing):

0:000> kL 100
ChildEBP RetAddr 
0014dbb4 77189254 ntdll!KiFastSystemCallRet
0014dbb8 75fec244 ntdll!ZwWaitForSingleObject+0xc
0014dc28 75fec1b2 kernel32!WaitForSingleObjectEx+0xbe
0014dc3c 72605389 kernel32!WaitForSingleObject+0x12
0014dc6c 726058e7 mscorwks!ClrWaitForSingleObject+0x24
0014e128 72608084 mscorwks!RunWatson+0x1df
0014e86c 7260874a mscorwks!DoFaultReportWorker+0xb59
0014e8a8 72657452 mscorwks!DoFaultReport+0xc3
0014e8cc 7265c0c7 mscorwks!WatsonLastChance+0x3f
0014e924 7265c173 mscorwks!CLRAddVectoredHandlers+0x209
0014e92c 7603f4be mscorwks!InternalUnhandledExceptionFilter+0x22
0014e9e8 771a85b7 kernel32!UnhandledExceptionFilter+0×127
0014e9f0 77139a14 ntdll!__RtlUserThreadStart+0×6f
0014ea04 771340f4 ntdll!_EH4_CallFilterFunc+0×12
0014ea2c 77189b99 ntdll!_except_handler4+0×8e
0014ea50 77189b6b ntdll!ExecuteHandler2+0×26
0014eb00 771899f7 ntdll!ExecuteHandler+0×24
0014eb00 03ca0141 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0014ee28 634c2f42 0×3ca0141
0014ee34 67715e44 System_ni+0×132f42
0014ee70 72431b4c System_ServiceProcess_ni+0×25e44
0014ee80 724421f9 mscorwks!CallDescrWorker+0×33
0014ef00 72456571 mscorwks!CallDescrWorkerWithHandler+0xa3
0014f03c 724565a4 mscorwks!MethodDesc::CallDescr+0×19c
0014f058 724565c2 mscorwks!MethodDesc::CallTargetWorker+0×1f
0014f070 724afac5 mscorwks!MethodDescCallSite::CallWithValueTypes+0×1a
0014f1d4 724af9e5 mscorwks!ClassLoader::RunMain+0×223
0014f43c 724aff35 mscorwks!Assembly::ExecuteMainMethod+0xa6
0014f90c 724b011f mscorwks!SystemDomain::ExecuteMainMethod+0×456
0014f95c 724b004f mscorwks!ExecuteEXE+0×59
0014f9a4 72f57c24 mscorwks!_CorExeMain+0×15c
0014f9b4 75fe4911 mscoree!_CorExeMain+0×2c
0014f9c0 7716e4b6 kernel32!BaseThreadInitThunk+0xe
0014fa00 7716e489 ntdll!__RtlUserThreadStart+0×23
0014fa18 00000000 ntdll!_RtlUserThreadStart+0×1b

We set exception context:

0:000> kv 100
ChildEBP RetAddr  Args to Child             
[...]
0014e9e8 771a85b7 0014ea18 77139a14 00000000 kernel32!UnhandledExceptionFilter+0×127 (FPO: [SEH])
[…]

0:000> .exptr 0014ea18

----- Exception record at 0014eb18:
ExceptionAddress: 03ca0141
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000000
Attempt to read from address 00000000

----- Context record at 0014eb34:
eax=00000001 ebx=08394ff8 ecx=00000000 edx=00000001 esi=056a2a94 edi=00000000
eip=03ca0141 esp=0014ee00 ebp=0014ee28 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246
03ca0141 3909            cmp     dword ptr [ecx],ecx  ds:0023:00000000=????????

Then we disassemble the code at crash point and it looks strange including calls through DS data segment:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> u 03ca0141
03ca0141 cmp     dword ptr [ecx],ecx
03ca0143 call    dword ptr ds:[36067C0h]
03ca0149 mov     ecx,dword ptr [esi+5Ch]
03ca014c cmp     dword ptr [ecx],ecx
03ca014e call    dword ptr ds:[3606D10h]
03ca0154 mov     dword ptr [ebp-1Ch],0
03ca015b mov     dword ptr [ebp-18h],0FCh
03ca0162 push    3CA0180h

However further disassembly finally reaches RET instruction:

0:000> u
03ca0167 jmp     03ca0169
03ca0169 movzx   edx,byte ptr [ebp-24h]
03ca016d mov     ecx,dword ptr [ebp-28h]
03ca0170 call    System_ServiceProcess_ni+0x25140 (67715140)
03ca0175 pop     eax
03ca0176 jmp     eax
03ca0178 lea     esp,[ebp-0Ch]
03ca017b pop     ebx

0:000> u
03ca017c pop     esi
03ca017d pop     edi
03ca017e pop     ebp
03ca017f ret
03ca0180 mov     dword ptr [ebp-18h],0
03ca0187 jmp     03ca0178
03ca0189 add     byte ptr [eax],al
03ca018b add     byte ptr [eax],al

and backward disassembling shows the matching function prolog code:

0:000> ub 03ca0141
03ca0127 movzx   eax,byte ptr [ebp-24h]
03ca012b test    eax,eax
03ca012d je      03ca0154
03ca012f cmp     dword ptr [esi+60h],0
03ca0133 je      03ca013e
03ca0135 mov     ecx,dword ptr [esi+60h]
03ca0138 call    dword ptr ds:[3C20010h]
03ca013e mov     ecx,dword ptr [esi+58h]

0:000> ub 03ca0127
03ca0114 push    esi
03ca0115 push    ebx
03ca0116 sub     esp,1Ch
03ca0119 xor     eax,eax
03ca011b mov     dword ptr [ebp-18h],eax
03ca011e mov     dword ptr [ebp-28h],ecx
03ca0121 mov     dword ptr [ebp-24h],edx
03ca0124 mov     esi,dword ptr [ebp-28h]

0:000> ub 03ca0114
03ca0102 retf
03ca0103 add     eax,dword ptr [eax+36h]
03ca0106 retf
03ca0107 add     ebx,dword ptr [esi+esi-35h]
03ca010b add     esi,esp
03ca010d cmp     eax,8B550360h
03ca0112 in      al,dx
03ca0113 push    edi

From stack trace I suspected this code as JIT-compiled .NET code of the the main assemebly method. And indeed I found the similar call signatures like mine

03ca0141 cmp     dword ptr [ecx],ecx
03ca0143 call    dword ptr ds:[36067C0h]

in the following MSDN article:

Drill Into .NET Framework Internals to See How the CLR Creates Runtime Objects

Hence the name of this pattern: JIT Code.

- Dmitry Vostokov @ DumpAnalysis.org -

OpenTask, iterative and incremental publisher, plans to release a guide for general computer users explaining crashes and hangs on personal computers ranging from mobile phones, netbooks and laptops to desktops and servers:

My Computer Crashes and Freezes: A Non-technical Guide to Software and Hardware Errors (ISBN: 978-19067171704)

- Dmitry Vostokov @ DumpAnalysis.org -

I recently became interested in who else has the same surname. Vostokov means Mr. East, east - ost (Ger.) - vostok (Rus.). Here is the brief list of people with links I’m going to expand periodically.

Alexander Vostokov - One of the first Russian philologists

Yevgeniy Vostokov - Composer

Sergei Vostokov - Mathematician

Viktor Vostokov - Author. “Secrets of East Healers”, “Lessons of Tibetan Medicine: The Art of Relaxation” and other books in Russian

Eugeny Vostokov - Artist

More to come.

- Dmitry Vostokov @ DumpAnalysis.org -

Part 2 briefly discussed debuggers and their commands. Debugger commands can be grouped together into scripts that can be run against memory dump files and their resulted textual output can be redirected to log files.

Therefore we start with the first step in our analysis process diagram introduced in Part 1:

Information Collection: Scripts

Here a script is a sequence of instructions to extract formatted textual information from a memory dump file (or a live system). There are many advantages of scripts and logs. Many companies, for example, banks, avoid sending plain memory dump files because of security considerations in order to prevent exposure of company or private information. After the advent of 64-bit personal computing complete memory dump files became larger and larger and it is now common to get 32Gb memory dumps. Processing multiple long-running commands can easily be done from textual log files. Additionally, scripts can be used to process hundreds of memory dumps in one go to look for similarities and differences. Many tools can be used for such purpose including built-in debugger scripting capabilities, shell scripting and regular expressions for intelligent search.

- Dmitry Vostokov @ DumpAnalysis.org -

“To” debug “is proper to man.”

François Rabelais, Gargantua and Pantagruel

- Dmitry Vostokov @ DumpAnalysis.org -

Consider this crash point:

0:000> r
eax=02d0f15c ebx=02a62918 ecx=77e41c30 edx=00000000 esi=ffffffff edi=02a8ed28
eip=76154193 esp=02d0f124 ebp=02d0f130 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
Application!GetData+0xb:
76154193 8b9eac000000    mov     ebx,dword ptr [esi+0ACh] ds:0023:000000ab=????????

Seeing 000000ab address we can think that ESI was 0 but it was 0xFFFFFFFF. Adding 0xAC to it produced an effective NULL data pointer 0xAB through integer addition overflow if we consider addition as unsigned. It is easy to see the result if we consider 0xFFFFFFFF as signed -1. Looking at stack trace and function disassembly we see that 0xFFFFFFFF was passed as a parameter:

0:000> kv
ChildEBP RetAddr  Args to Child             
02d0f130 7616328d ffffffff 02d0f15c 02d0f150 Application!GetData+0xb
[…]
02d0ffec 00000000 740420d8 02a74070 00000000 kernel32!BaseThreadStart+0×34

0:000> u Application!GetData
Application!GetData:
76154188 mov     edi,edi
7615418a push    ebp
7615418b mov     ebp,esp
7615418d push    ecx
7615418e push    ebx
7615418f push    esi
76154190 mov     esi,dword ptr [ebp+8]
76154193 mov     ebx,dword ptr [esi+0ACh]

This is an example of a sentinel pointer marking the end of a linked list, for example, although NULL pointers having 0 value are usually used. Also -1 value can be used to assign an invalid pointer value. 

- Dmitry Vostokov @ DumpAnalysis.org -

When looking at a stack trace of one crashed process we noticed an invalid code pointer. It is not a NULL code pointer but has the same stack trace pattern:

0:000> kL
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0013dfb4 00401791 0×5e388583
0013fdf4 0040189f Application!RequestData+0×3e1
0013fee4 00401d0a Application!main+0×3f
0013ffc0 77e4f23b Application!mainCRTStartup+0×16c
0013fff0 00000000 kernel32!BaseProcessStart+0×23

When we look at raw stack data and examine the backward disassembly of the return address we see that invalid code was called from RequestData function and WinDbg stack trace reconstruction is not suspicious (it is structurally and semantically correct):

0:000> dds esp l10
0013dfb8  00401791 Application!RequestData+0x3e1
0013dfbc  00000140
0013dfc0  0013ee50
0013dfc4  00000fa4
0013dfc8  00000000
0013dfcc  00000000
0013dfd0  00000ece
0013dfd4  0013ffc0
0013dfd8  7ffdc000
0013dfdc  00000140
0013dfe0  0000054c
0013dfe4  50000002
0013dfe8  4b4919ac
0013dfec  00000000
0013dff0  00000000
0013dff4  003f003c

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> ub 00401791
Application!RequestData+0x3c8:
00401778 Application!RequestData+0x3d0 (00401780)
0040177a lea     ebx,[ebx]
00401780 push    0
00401782 push    eax
00401783 lea     ecx,[esp+esi+0E30h]
0040178a push    ecx
0040178b push    edi
0040178c call    Application!recv (0040e382)

When seeing recv call we might suspect that the crash happened just inside that function because the raw stack data upwards (lower addresses) doesn’t have any execution residue left from nested function calls:

0:000> dds esp-100 esp
0013deb8  00000000
0013debc  00000000
0013dec0  00000000
0013dec4  00000000
0013dec8  00000000
0013decc  00000000
0013ded0  00000000
0013ded4  00000000
0013ded8  00000000
0013dedc  00000000
0013dee0  00000000
0013dee4  00000000
0013dee8  00000000
0013deec  00000000
0013def0  00000000
0013def4  00000000
0013def8  00000000
0013defc  00000000
0013df00  00000000
0013df04  00000000
0013df08  00000000
0013df0c  00000000
0013df10  00000000
0013df14  00000000
0013df18  00000000
0013df1c  00000000
0013df20  00000000
0013df24  00000000
0013df28  00000000
0013df2c  00000000
0013df30  00000000
0013df34  00000000
0013df38  00000000
0013df3c  00000000
0013df40  00000000
0013df44  00000000
0013df48  00000000
0013df4c  00000000
0013df50  00000000
0013df54  00000000
0013df58  00000000
0013df5c  00000000
0013df60  00000000
0013df64  00000000
0013df68  00000000
0013df6c  00000000
0013df70  00000000
0013df74  00000000
0013df78  00000000
0013df7c  00000000
0013df80  00000000
0013df84  00000000
0013df88  00000000
0013df8c  00000000
0013df90  00000000
0013df94  00000000
0013df98  00000000
0013df9c  00000000
0013dfa0  00000000
0013dfa4  00000000
0013dfa8  00000000
0013dfac  00000000
0013dfb0  00000000
0013dfb4  00000000
0013dfb8  00401791 Application!RequestData+0x3e1

So we follow recv call forward disassembly (notice that the first jump is indirect):

0:000> u 0040e382
Application!recv:
0040e382 jmp     dword ptr [Application!_imp__recv (00410180)]
Application!closesocket:
0040e388 jmp     dword ptr [Application!_imp__closesocket (00410170)]
Application!WSAGetLastError:
0040e38e jmp     dword ptr [Application!_imp__WSAGetLastError (00410174)]
Application!send:
0040e394 jmp     dword ptr [Application!_imp__send (00410178)]
Application!connect:
0040e39a jmp     dword ptr [Application!_imp__connect (0041017c)]
Application!htons:
0040e3a0 jmp     dword ptr [Application!_imp__htons (00410198)]
Application!setsockopt:
0040e3a6 jmp     dword ptr [Application!_imp__setsockopt (00410184)]
Application!socket:
0040e3ac jmp     dword ptr [Application!_imp__socket (00410188)]

0:000> dps 00410180 l10
00410180  71ad2f7f ws2_32!recv
00410184  71ad2d47 ws2_32!setsockopt
00410188  71ad410c ws2_32!socket
0041018c  71ad7ca1 ws2_32!gethostbyname
00410190  71ad4f3b ws2_32!WSAStartup
00410194  71ad7b5b ws2_32!gethostname
00410198  71ad28bc ws2_32!htons
0041019c  71ad3da8 ws2_32!WSACleanup
004101a0  00000000
004101a4  00000000
004101a8  00000000
004101ac  00000000
004101b0  00000000
004101b4  45cd184e
004101b8  00000000
004101bc  00000002

0:000> u 71ad2f7f
ws2_32!recv:
71ad2f7f jmp     7fd60000
71ad2f84 sub     esp,10h
71ad2f87 push    ebx
71ad2f88 xor     ebx,ebx
71ad2f8a cmp dword ptr [ws2_32!PrologPointer (71ae4044)],offset ws2_32!Prolog_v2 (71ad6067)
71ad2f94 push    esi
71ad2f95 je      ws2_32!recv+0×18 (71ad6207)
71ad2f9b lea     eax,[ebp-8]

0:000> u 7fd60000
7fd60000 jmp     DllA!recv_patch (612101b6)
7fd60005 mov     edi,edi
7fd60007 push    ebp
7fd60008 mov     ebp,esp
7fd6000a jmp     ws2_32!recv+0×5 (71ad2f84)
7fd6000f add     byte ptr [eax],al
7fd60011 add     byte ptr [eax],al
7fd60013 add     byte ptr [eax],al

Finally we see that sockets library functions were patched by a 3rd-party module DllA and we need to contact its vendor.

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced memory dump analysis certification will have x86/x64-based tracks for Windows and Unix (including Linux / FreeBSD / Mac OS X). Each track consists of 3 exams, each having its own set of requirements and scope:

• Fundamentals of Memory Dump Analysis
• Intermediate Memory Dump Analysis
• Advanced Memory Dump Analysis

More information will be available later. The initiative is supported by OpenTask.

- Dmitry Vostokov @ DumpAnalysis.org -

This is an example from the sample question set of the forthcoming memory dump analysis certification planned by Memory Analysis and Debugging Institute. You can reply to this post with your answers.

Q. Interpret the fragment from WinDbg output below.

0:002> !teb
[...]
TEB at 7efaf000
[...]

0:002> dds poi(7efaf000+8) poi(7efaf000+4)
01192000  00000000
01192004  00000000
01192008  00000000
[...]
0119e448  00010020
0119e44c  00030002
0119e450  00050004
0119e454  00070006
0119e458  00090008
0119e45c  000b000a
0119e460  000d000c
0119e464  000f000e
0119e468  00110010
0119e46c  00130012
0119e470  00150014
0119e474  00170016
0119e478  00190018
0119e47c  001b001a
0119e480  001d001c
0119e484  001f001e
0119e488  00210020
0119e48c  00230022
0119e490  00250024
0119e494  00270026
0119e498  00290028
0119e49c  002b002a
0119e4a0  002d002c
0119e4a4  002f002e
0119e4a8  00310030
0119e4ac  00330032
0119e4b0  00350034
0119e4b4  00370036
0119e4b8  00390038
0119e4bc  003b003a
0119e4c0  003d003c
0119e4c4  003f003e
0119e4c8  00410040 MyService!__InternalCxxFrameHandler+0x5c
0119e4cc  00430042
0119e4d0  00450044
0119e4d4  00470046
0119e4d8  00490048
0119e4dc  004b004a
0119e4e0  004d004c
0119e4e4  004f004e
0119e4e8  00510050 advapi32!`string'+0x164
0119e4ec  00530052 advapi32!GetPerflibKeyValue+0x184
0119e4f0  00550054 advapi32!`string'+0x20c
0119e4f4  00570056 advapi32!_NULL_IMPORT_DESCRIPTOR+0x2714
0119e4f8  00590058 advapi32!szPerflibSectionName <PERF> (advapi32+0x90058)
0119e4fc  005b005a shlwapi!_CRT_INIT+0xaf
0119e500  005d005c shlwapi!_OpenProgidKey+0xee
0119e504  005f005e shlwapi!_pRawDllMain <PERF> (shlwapi+0x5005e)
0119e508  00410060 MyService!__InternalCxxFrameHandler+0x7c
0119e50c  00430042
0119e510  00450044
0119e514  00470046
0119e518  00490048
0119e51c  004b004a
0119e520  004d004c
0119e524  004f004e
0119e528  00510050 advapi32!`string'+0x164
0119e52c  00530052 advapi32!GetPerflibKeyValue+0x184
0119e530  00550054 advapi32!`string'+0x20c
0119e534  00570056 advapi32!_NULL_IMPORT_DESCRIPTOR+0x2714
0119e538  00590058 advapi32!szPerflibSectionName <PERF> (advapi32+0x90058)
0119e53c  007b005a
0119e540  007d007c
0119e544  007f007e
[...]

Choose all answers that are valid:

a. ASCII string fragment
b. Raw stack data
c. Exception handling
d. UNICODE string fragment
e. Partial stack traces
f. Performance monitoring
g. Execution residue

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes the script featured in part 1 doesn’t work because of the lack of symbols or something else:

***                                                                  
***                                                                  
*** Your debugger is not using the correct symbols    
*** 
*** In order for this command to work properly, your symbol path
*** must point to .pdb files that have full type information.
*** 
*** Certain .pdb files (such as the public OS symbols) do not
*** contain the required information.  Contact the group that
*** provided you with these symbols if you need this command to
*** work.
*** 
*** Type referenced: ntdll!_NT_TIB
***
Couldn’t resolve error at ‘ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2′
                    ^ Extra character error in ‘~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2′

This is the case where !teb WinDbg command doesn’t work and we can cope with this as shown in the following post:

Coping with missing symbolic information

Therefore we can adjust our user-mode script to use hard-coded offsets and delineate raw stack outputs by the output of kv WinDbg command:

~*e r? $t0 = @$teb; r? $t1 = @$t0+8; r? $t2 = @$t0+4; kv 100; dps poi(@$t1) poi(@$t2)

- Dmitry Vostokov @ DumpAnalysis.org -

A server was hanging and a complete memory dump was generated. Usual checks for resource contention or CPU spikes like !locks and !running WinDbg commands didn’t reveal anything. Therefore we decided to list all threads in the system using !stacks and !process 0 ff commands. The former command gives a birds eye overview of threads and it was found suspicious that many threads in different processes had 0 ticks waiting and had this top level call:

0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]

[88d9cd88 explorer.exe]
Proc.Thread .Thread  Ticks   ThreadState Blocker
1ecc.001eb4 88d99db0 0000000 Blocked     3rdPartyAVDriver+0×1394
[…]

[862bad88 userinit.exe]
4394.005570 878f0790 0000000 Blocked     3rdPartyAVDriver+0×1394

Inspection of the latter command detailed output of thread stacks showed that all these suspicious threads were in fact blocked when loading a DLL:

0: kd> .process /r /p 862bad88
Implicit process is now 862bad88
Loading User Symbols

0: kd> .thread 878f0790
Implicit thread is now 878f0790

0: kd> kL 100
ChildEBP RetAddr 
9aeb529c 80833e95 nt!KiSwapContext+0x26
9aeb52c8 80829027 nt!KiSwapThread+0x2e5
9aeb5310 b8dff394 nt!KeDelayExecutionThread+0x2ab
WARNING: Stack unwind information not available. Following frames may be wrong.
9aeb5338 b8dffe35 3rdPartyAVDriver+0×1394
9aeb5354 b8e00208 3rdPartyAVDriver+0×1e35
9aeb5398 b8e0e45a 3rdPartyAVDriver+0×2208
9aeb53d0 8081e087 3rdPartyAVDriver+0×1045a
9aeb53e4 b7d6e41b nt!IofCallDriver+0×45
9aeb53f4 b7d6a3ee DriverA!Dispatch+0xfe
9aeb5404 8081e087 DriverA!KUI_dispatch+0×6d
9aeb5418 b8bdb4e1 nt!IofCallDriver+0×45
9aeb543c b8bdb755 DriverB!PassThrough+0xd3
9aeb545c 8081e087 DriverB!Dispatch+0×153
9aeb5470 b6a7b81d nt!IofCallDriver+0×45
9aeb5498 b6a6c5bb DriverC!LowerPassThrough+0×48
9aeb54bc b6a6cc32 DriverC+0×75bb
9aeb5554 b6a7aa57 DriverC+0×7c32
9aeb5564 b6a7aaa7 DriverC+0×15a57
9aeb558c 8081e087 DriverC!DispatchPassThrough+0×48
9aeb55a0 808fb13b nt!IofCallDriver+0×45
9aeb5688 80939c6a nt!IopParseDevice+0xa35
9aeb5708 80935d9e nt!ObpLookupObjectName+0×5b0
9aeb575c 808ece57 nt!ObOpenObjectByName+0xea
9aeb57d8 808ee0f1 nt!IopCreateFile+0×447
9aeb5834 808f1e31 nt!IoCreateFile+0xa3
9aeb5874 8088b19c nt!NtOpenFile+0×27
9aeb5874 7c94860c nt!KiFastCallEntry+0xfc
0007e1fc 7c947339 ntdll!KiFastSystemCallRet
0007e200 7c952653 ntdll!NtOpenFile+0xc
0007e278 7c9525db ntdll!LdrpCreateDllSection+0×4e
0007e364 7c95425c ntdll!LdrpMapDll+0×297
0007e610 7c95408d ntdll!LdrpLoadDll+0×1e4
0007e88c 7c801bf7 ntdll!LdrLoadDll+0×198
0007e8f4 7c81c70b kernel32!LoadLibraryExW+0×1b2
0007e908 7c93a352 kernel32!LoadLibraryW+0×11
0007ee38 7c95348d ntdll!LdrpCallInitRoutine+0×14
0007ef40 7c954339 ntdll!LdrpRunInitializeRoutines+0×367
0007f1d4 7c95408d ntdll!LdrpLoadDll+0×3cd
0007f450 7c801bf7 ntdll!LdrLoadDll+0×198
0007f4b8 7c81c70b kernel32!LoadLibraryExW+0×1b2
0007f4cc 77e556fb kernel32!LoadLibraryW+0×11
0007f558 77e3dc66 USER32!LoadAppDlls+0×179
0007f694 77e3df10 USER32!ClientThreadSetup+0×1c4
0007f698 7c948556 USER32!__ClientThreadSetup+0×5
0007f698 80831348 ntdll!KiUserCallbackDispatcher+0×2e
9aeb5b4c 8091f9e1 nt!KiCallUserMode+0×4
9aeb5ba4 bf8bc7c3 nt!KeUserModeCallback+0×8f
9aeb5bcc bf8bbc93 win32k!xxxClientThreadSetup+0×21
9aeb5ccc bf8b6c46 win32k!xxxCreateThreadInfo+0×804
9aeb5cdc bf8b7ae4 win32k!UserThreadCallout+0×71
9aeb5cf8 80947a1b win32k!W32pThreadCallout+0×3a
9aeb5d54 8088ae9e nt!PsConvertToGuiThread+0×153
9aeb5d58 00000000 nt!KiBBTUnexpectedRange+0xc

The functions shown in blue are known from past issues to be pass through forwarding IRP to the lower drivers in a device driver stack.

Actually the thread stack above belongs to the main thread blocked while loading application DLLs. The conclusion was to check 3rdPartyAVDriver.sys vendor.

- Dmitry Vostokov @ DumpAnalysis.org -

As soon as I wrote my review of the 2nd edition I found out that the 3rd edition was recently published and immediately bought it. I intend to read it from cover to cover again and publish my notes and comments in my reading notebook on Software Generalist blog. The new edition is also bundled with a companion CD.

Programming Language Pragmatics, Third Edition

Hope in one of subsequent editions the author includes my Riemann Programming Language :-)

- Dmitry Vostokov @ DumpAnalysis.org -

My local browser crashed today when I did copy-paste of an RTF text into an HTML editor window. The dump was not saved because I previously set up logging as described here (my script doesn’t include .dump commands):

All at once: postmortem logs and dump files

Looking at stack trace I noticed that the crash happened during HTML processing (call arguments are removed for visual clarity):

STACK_TEXT: 
0476de3c 6970d597 html!FPseudoStyleBis+0x26
0476de48 69703b0e html!BisFromLpxszStyle+0x1c
0476de60 69702ba9 html!LwMultDivRU+0x4b6
0476dea4 6970518a html!FMarkListCallback+0x56c
0476deb4 697068b7 html!JcCalcFromXaExtents+0x91
0476df60 697070c5 html!EmitNonBreakingSpace+0x445
0476e08c 697107ff html!FEmitHtmlFnOtag+0x17d
0476e0b0 696ec6a8 html!ConvertRtfToForeign+0x105
0476e538 696ec745 html!FceRtfToForeign+0x266
0476e560 6b7e5ad4 html!RtfToForeign32+0x51
0476e9a8 6b7e5c83 mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml+0x163
0476edfc 6b79cc15 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x11a
0476ee18 6b79cd81 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x38
0476ee2c 6b7cdcea mshtmled!CHTMLEditor::ConvertRTFToHTML+0x12
0476ee98 6b7ce392 mshtmled!CPasteCommand::PasteFromClipboard+0x2c0
0476ef08 6b78d218 mshtmled!CPasteCommand::PrivateExec+0x47a
0476ef2c 6b78d1ad mshtmled!CCommand::Exec+0x4b
0476ef50 6b470d14 mshtmled!CMshtmlEd::Exec+0xf9
0476ef80 6b4688a8 mshtml!CEditRouter::ExecEditCommand+0xd6
0476f328 6b5ceccf mshtml!CDoc::ExecHelper+0x338d
0476f374 6b468a2f mshtml!CFrameSite::Exec+0x264
0476f3a8 6b4687af mshtml!CDoc::RouteCTElement+0xf1
0476f740 6b468586 mshtml!CDoc::ExecHelper+0x325e
0476f760 6b510e7b mshtml!CDoc::Exec+0x1e
0476f798 6b48a708 mshtml!CDoc::OnCommand+0x9c
0476f8ac 6b3997e1 mshtml!CDoc::OnWindowMessage+0x841
0476f8d8 766ff8d2 mshtml!CServer::WndProc+0x78
0476f904 766ff794 USER32!InternalCallWinProc+0x23
0476f97c 767006f6 USER32!UserCallWinProcCheckWow+0x14b
0476f9ac 7670069c USER32!CallWindowProcAorW+0x97
0476f9cc 6ce1851b USER32!CallWindowProcW+0x1b
WARNING: Stack unwind information not available. Following frames may be wrong.
0476fa40 6ce0cdc6 GoogleToolbarDynamic_6D0D6FD66D664927!DllGetClassObject+0x24981
0476fa64 6ce9beaa GoogleToolbarDynamic_6D0D6FD66D664927!DllGetClassObject+0x1922c
0476fa94 766ff8d2 GoogleToolbarDynamic_6D0D6FD66D664927!DllGetClassObject+0xa8310
0476fac0 766ff794 USER32!InternalCallWinProc+0x23
0476fb38 76700a05 USER32!UserCallWinProcCheckWow+0x14b
0476fb78 76700afa USER32!SendMessageWorker+0x4b7
0476fb98 6b47fb9b USER32!SendMessageW+0x7c
0476fbc4 6b3d8e5a mshtml!CElement::PerformTA+0x71
0476fbe4 6b3d8db9 mshtml!CDoc::PerformTA+0xd8
0476fc60 6b46381c mshtml!CDoc::PumpMessage+0x8e0
0476fd14 6b463684 mshtml!CDoc::DoTranslateAccelerator+0x33f
0476fd30 6b4634cc mshtml!CServer::TranslateAcceleratorW+0x56
0476fd50 70c9f550 mshtml!CDoc::TranslateAcceleratorW+0x83
0476fd6c 70c9f600 IEFRAME!CProxyActiveObject::TranslateAcceleratorW+0x30
0476fd90 70c9fca1 IEFRAME!CDocObjectView::TranslateAcceleratorW+0xb1
0476fdb0 70c9faf4 IEFRAME!CCommonBrowser::v_MayTranslateAccelerator+0xda
0476fddc 70c9f7b0 IEFRAME!CShellBrowser2::_MayTranslateAccelerator+0x68
0476fdec 70c9f7f5 IEFRAME!CShellBrowser2::v_MayTranslateAccelerator+0x15
0476fe58 76894911 IEFRAME!CTabWindow::_TabWindowThreadProc+0x264
0476fe64 776ae4b6 kernel32!BaseThreadInitThunk+0xe
0476fea4 776ae489 ntdll!__RtlUserThreadStart+0x23
0476febc 00000000 ntdll!_RtlUserThreadStart+0x1b

I immediately recalled that in Andreas Zeller’s book Why Programs Fail a browser parsing HTML was used as an example to show delta debugging (I think it was Mozilla).

The complete log file can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org -

A typical trace is a detailed narrative. It is accompanied by a problem description that lists essential facts. Therefore the first task of any trace analysis is to check the presence of Basic Facts in the trace. If they are not visible or do not correspond then the trace was possibly not recorded during the problem or was taken from a different computer or under different conditions. Here is an example. A user “test01″ cannot connect to an application. We look at the trace and find this statement:

No   PID  TID  Date      Time         Statement
[...]
3903 3648 5436 4/29/2009 16:17:36.150 User Name: test01
[...]

At least we can be sure that this trace was taken for the user test01 especially when we expect this or similar trace statements. If we could not see this trace statement we can suppose that the trace was taken at the wrong time, for example, after the problem happened already.

- Dmitry Vostokov @ TraceAnalysis.org -

This is the enhanced version of Dump2Wave technology that allows to transform computational operations into audible artifacts.

Computational processes and threads are fiber bundled with native memory visualization techniques to create audio and visual images of powerful memory topoi. This opens the new era in music. The closure of analog -> digital -> analog enables visualization and auralization of finite and infinite (transfinite) digital data.

Stay tuned! More on this later…

- Dmitry Vostokov @ DumpAnalysis.org -

The community of Citrix and Microsoft Terminal Services / Virtualization meets again in Dublin, Ireland:

Event: PubForum 2009 Dublin - just one this year
Date: 5 to 7 June 2009
Where: Dublin, Ireland
Event Location: Camden Court Hotel, Dublin
Sessions: 20
Master Classes: 6
Visitors: 75 - 80
Attendee Fee: just 199 EUR

Agenda of the Event

- Dmitry Vostokov @ DumpAnalysis.org -

On universal memory dumps:

“[…] the first man who noticed the analogy between a” dump “and” an observation “made a notable advance in the history of thought.”

Alfred North Whitehead, Science and the Modern World

- Dmitry Vostokov @ DumpAnalysis.org -

“The first rule of” debugging “is to have brains and good luck. The second rule of” debugging “is to sit tight and wait till you” hit “a” breakpoint.

George Pólya, How to Solve It

- Dmitry Vostokov @ DumpAnalysis.org -

Urstoff of Anaximenes is Air much like Memory in memoidealism. How do concrete objects develop from invisible Air? Through the process of condensation and rarefaction, quality arises from quantity (reduction process). Eternity of Urstoff is one of the main features of Milesian philosophers and memory religion. No additional worlds are possible in their philosophies. They are “materialists” because of their material Urstoff. Memoidealists are “idealists” because of their ideal notion of Memory.

- Dmitry Vostokov @ DumpAnalysis.org -