Archive for October, 2009

DebugWare Patterns: A Case Study (Part 1)

Friday, October 30th, 2009

Suppose we want to write a support tool to troubleshoot a frequent software problem. Would we start with GUI? Most tools are command line tools and we therefore choose Tool Façade pattern to decouple GUI from CLI:

By making our GUI optional we postpone its development until later implementation phase when we want to beautify or simplify the tool user interface. The troubleshooting of software problem we want to automate is quite complex, scattered across several knowledge base articles, with many troubleshooting steps and various checks. We therefore treat different steps as TUWs, Troubleshooting Units of Work (aggregating related ones if necessary) and choose Checklist pattern to coordinate them:

Initially we think that just one checklist is suffice but while studying the domain of troubleshooting we find that the problem usually spans client and server computers, where servers sometimes have client roles as well and different troubleshooting steps and checks are needed. In order to simplify the tool we apply Span Differentiator pattern and our architectural UML component diagram becomes more refined:

We also see that different TUW use similar operations like querying registry values or checking whether certain processes are running and factor them out into separate SMs, Supporting Modules:

Deployment methods are important in troubleshooting where it is not always desirable to change configuration on production servers and we plan for optional Self Extractor pattern:

In diagrams we liberally mixed component and static UML diagramming elements. We also need to design a uniform interface that each TUW module exposes to the main tool coordinator module but we leave it for the part 2.

- Dmitry Vostokov @ -

My Failed Job Interviews

Thursday, October 29th, 2009

As a dual to my previous Resume and CV: As a Book (it actually has a “bug” on the cover, try to spot it) I plan to publish the long time memories in the following book next year:

My Failed Job Interviews: Reflections on 50 Percent (ISBN: 978-1906717889)

The recollections of my pre-memory dump analysis era span East and West, small and giant software companies, full time and part time, office and remote job positions, direct and recruitment company hiring, phone and on-site, technical and business interviews.

- Dmitry Vostokov @ -

DebugWare Patterns (Part 12)

Wednesday, October 28th, 2009

Support tools can be packaged for delivery as (not counting optional separate readme and EULA files):

  • - A raw collection of modules
  • - A single installation package
  • - A single archived file
  • - A single self-extractable component

The latter common packaging choice is a pattern itself that I call Self Extractor. Typical examples include sysinternals tools like Process Monitor. Here is a simple unorthodox UML diagram for a single platform tool combining elements from static and component diagrams to illustrate component embedding (not sharing):

SM is a supporting module, for example, a driver, to carry out some task or receive notifications.

Some tools even extract different components depending on the processor architecture, for example, x86 and x64. This is illustrated on a more elaborate UML diagram:

The typical example would be a backward compatible x86 Extractor that extracts an x64 driver (Extractee) for an x64 OS kernel that doesn’t load x86 drivers.

- Dmitry Vostokov @ -

Crash Dump Analysis Patterns (Part 90)

Wednesday, October 28th, 2009

Sometimes we have a managed code exception that was enveloping a handled unmanaged code exception, Mixed (Nested) Exception:

0:000> !analyze -v


EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.



EXCEPTION_OBJECT: !pe 1f9af1ac
Exception object: 1f9af1ac
Exception type: System.AccessViolationException
Message: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    0012EF3C 28DD9AF9 DllA!Component.getFirstField()+0×11
    0012EFC8 7B194170 System_Windows_Forms_ni!System.Windows.Forms.Control.OnClick(System.EventArgs)+0×70
    0012EFE0 7B6F74B4 System_Windows_Forms_ni!System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)+0×170
    0012F06C 7BA29B66 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0×861516
    0012F0C4 7B1D1D6A System_Windows_Forms_ni!System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Message ByRef)+0×2a
    0012F0D0 7B1C8640 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)+0×10
    0012F0D8 7B1C85C1 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0×31
    0012F0EC 7B1C849A System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0×5a


We see that it was the access violation exception and check the thread with TID cfc:

0:000> kL
ChildEBP RetAddr 
0012db54 77d70dde ntdll!KiFastSystemCallRet
0012db58 7b1d8e48 user32!NtUserWaitMessage+0xc
0012dbec 7b1d8937 System_Windows_Forms_ni+0x208e48
0012dc44 7b1d8781 System_Windows_Forms_ni+0x208937
0012dc74 7b6edd1f System_Windows_Forms_ni+0x208781
0012dc8c 7b72246b System_Windows_Forms_ni+0x71dd1f
0012dd18 7b722683 System_Windows_Forms_ni+0x75246b
0012dd58 7b6f77f6 System_Windows_Forms_ni+0x752683
0012dd64 7b6fa27c System_Windows_Forms_ni+0x7277f6
0012f148 77d6f8d2 System_Windows_Forms_ni+0x72a27c
0012f174 77d6f794 user32!InternalCallWinProc+0x23
0012f1ec 77d70008 user32!UserCallWinProcCheckWow+0x14b
0012f250 77d70060 user32!DispatchMessageWorker+0x322
0012f260 0a1412fa user32!DispatchMessageW+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012f27c 578439f7 0xa1412fa
0012f2ec 578430c9 WindowsBase_ni+0x939f7
0012f2f8 5784306c WindowsBase_ni+0x930c9
0012f304 55bed46e WindowsBase_ni+0x9306c
0012f310 55bec76f PresentationFramework_ni+0x1cd46e
0012f334 55bd3aa6 PresentationFramework_ni+0x1cc76f

If there was an exception it must be hidden so we inspect the thread raw stack:

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0012e470
    StackBase:            00130000
    StackLimit:           0011e000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             00000b6c . 00000cfc
    RpcHandle:            00000000
    Tls Storage:          7ffdf02c
    PEB Address:          7ffd4000
    LastErrorValue:       0
    LastStatusValue:      c0000139
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dps 0011e000 00130000
0011e000  00000000
0011e004  00000000
0011e008  00000000
0012e72c  00130000
0012e730  0011e000
0012e734  00ee350d
0012e738  0012ea3c
0012e73c  77f299f7 ntdll!KiUserExceptionDispatcher+0xf
0012e740  0012e750
0012e744  0012e76c
0012e748  0012e750
0012e74c  0012e76c
0012e750  c0000005
0012e754  00000000
0012e758  00000000
0012e75c  77f17d89 ntdll!RtlLeaveCriticalSection+0×9
0012e760  00000002
0012e764  00000001
0012e768  00000028
0012e76c  0001003f
0012e770  00000000
0012e774  00000000
0012e778  00000000
0012e77c  00000000

0:000> .cxr 0012e76c
eax=00000020 ebx=09ca1fa0 ecx=781c1b78 edx=00000001 esi=00000020 edi=09ca1ff8
eip=77f17d89 esp=0012ea38 ebp=0012ea3c iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202
77f17d89 834608ff        add     dword ptr [esi+8],0FFFFFFFFh ds:0023:00000028=????????

0:000> kL
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
0012ea3c 7813e5b5 ntdll!RtlLeaveCriticalSection+0x9
0012ea44 2071c9ba msvcr80!_unlock_file+0x35
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ea68 2071c31e DllB!getType+0×286a
0012ee34 206bbfbc DllB!getType+0×3eb
0012ee68 206c8abb DllC+0xbfbc
0012ee98 79e71ca7 DllC!getFirstField+0×3b

0012f148 77d6f8d2 mscorwks!NDirectGenericStubReturnFromCall
0012f1ec 77d70008 user32!InternalCallWinProc+0×23
0012f240 77db51b9 user32!DispatchMessageWorker+0×322
0012f4a4 79e95feb user32!_W32ExceptionHandler+0×18
0012f4fc 79e968b0 mscorwks!MetaSig::HasRetBuffArg+0×5
0012f50c 79e9643e mscorwks!MetaSig::MetaSig+0×3a
0012f610 79e96534 mscorwks!MethodDesc::CallDescr+0xaf
0012f62c 79e96552 mscorwks!MethodDesc::CallTargetWorker+0×1f
0012f644 79eefa45 mscorwks!MethodDescCallSite::CallWithValueTypes+0×1a
0012f7a8 79eef965 mscorwks!ClassLoader::RunMain+0×223

Therefore we identified DllB and DllC components as suspicious. If we check exception chain we see that .NET runtime registered custom exception handlers:

0:000> !exchain
0012e470: mscorwks!COMPlusNestedExceptionHandler+0 (79edd6d7)
0012f13c: mscorwks!FastNExportExceptHandler+0 (7a00a2e7)

0012f1dc: user32!_except_handler4+0 (77db51ba)
0012f240: user32!_except_handler4+0 (77db51ba)
0012f46c: mscorwks!COMPlusFrameHandler+0 (79edc3bc)
0012f4c0: mscorwks!_except_handler4+0 (79f908a2)
0012f798: mscorwks!_except_handler4+0 (79f908a2)
0012fa04: mscorwks!GetManagedNameForTypeInfo+a680 (7a328d90)
0012fed4: mscorwks!GetManagedNameForTypeInfo+82c8 (7a325a3a)
0012ff20: mscorwks!_except_handler4+0 (79f908a2)
0012ff6c: mscorwks!GetManagedNameForTypeInfo+a6e (7a319ee4)
0012ffc4: ntdll!_except_handler4+0 (77ed9834)
Invalid exception stack at ffffffff

We check that GetManagedNameForTypeInfo+a6e (7a319ee4) is an exception handler indeed:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> uf 7a319ee4
78158aeb push ebp
78158aec mov ebp,esp
78158aee sub esp,8
78158af1 push ebx
78158af2 push esi
78158af3 push edi
78158af4 cld
78158af5 mov dword ptr [ebp-4],eax
78158af8 xor eax,eax
78158afa push eax
78158afb push eax
78158afc push eax
78158afd push dword ptr [ebp-4]
78158b00 push dword ptr [ebp+14h]
78158b03 push dword ptr [ebp+10h]
78158b06 push dword ptr [ebp+0Ch]
78158b09 push dword ptr [ebp+8]
78158b0c call msvcr80!__InternalCxxFrameHandler (7815897e)
78158b11 add esp,20h
78158b14 mov dword ptr [ebp-8],eax
78158b17 pop edi
78158b18 pop esi
78158b19 pop ebx
78158b1a mov eax,dword ptr [ebp-8]
78158b1d mov esp,ebp
78158b1f pop ebp
78158b20 ret

79f5f258 jmp dword ptr [mscorwks!_imp____CxxFrameHandler3 (79e711c4)]

7a319ee4 mov edx,dword ptr [esp+8]
7a319ee8 lea eax,[edx+0Ch]
7a319eeb mov ecx,dword ptr [edx-30h]
7a319eee xor ecx,eax
7a319ef0 call mscorwks!__security_check_cookie (79e72037)
7a319ef5 mov eax,offset mscorwks!_CT??_R0H+0xc14 (7a319f00)
7a319efa jmp mscorwks!__CxxFrameHandler3 (79f5f258)

- Dmitry Vostokov @ -

Bugtation No.107

Tuesday, October 27th, 2009

“Simplifying computing with a” crash.

Dmitry Vostokov’s excavative exclamation upon hearing a marketing slogan “I have a crush on you”

- Dmitry Vostokov @ -

DNA and RNA of Ruptured Computation

Friday, October 23rd, 2009

First, I introduce the notion of a ruptured computation that is a relaunched, restarted or resumed process from a checkpoint after a rupture in its fabric, for example, crash, hang, spike and leak forcing termination. DNA can be an abbreviation of a Dump of Nasty Application or Double-ruptures of Nasty Application but I got an enlightenment spike in my head recently after recognizing that C, H, S and L are 4 DNA bases: Crash, Hang, Spike and Leak. Do you see a parallel with the real DNA and its 4 bases: A, T, G, C? Now a sample of a  ruptured computation DNA strand (imagine, we launch an application and it crashes, …, crashes, …, crashes, …, spikes, …, crashes, …, hangs, …) can be represented as: …CCCSCH… . What about its second strand? I’m in search of it now but in the mean time I introduce the second notion of RNA (Really Nasty Application or Ruptures of Nasty Application) that is just one strand of …CCCSCH… as needed (real RNA is mostly single-stranded).

- Dmitry Vostokov @ -

Crash Dump Analysis Patterns (Part 89)

Friday, October 23rd, 2009

When looking at stack traces seasoned memory dump analysis and software maintenance engineers immediately spot the right function to dig into its source code:

05b3f514 66ed52d3 08d72fee 08da6fc4 05b3f540 msvcrt!wcscpy+0xe
05b3f53c 77c50193 07516fc8 06637fc4 00000006 ModuleA!Add+0xd8
05b3f568 77cb33e1 66ed51fb 05b3f750 00000006 rpcrt4!Invoke+0x30
05b3f968 77cb1968 08d6afe0 0774cf4c 0660af28 rpcrt4!NdrStubCall2+0x299

Most will start with ModuleA!Add and examine parameters to wcscpy. This is because wcscpy (UNICODE version of strcpy) is considered as a Well-Tested Function. For the purposes of default analysis via !analyse -v command it is possible to configure WinDbg to ignore our own functions and modules as well if we are sure they were well-tested or pass-through. For details please see the old minidump analysis case study.

- Dmitry Vostokov @ -

Crash Dump Analysis Patterns (Part 88)

Friday, October 23rd, 2009

Some modules like drivers or runtime DLLs are always present after some action has happened. I call them Effect Components. It is the last thing to assume them to be the “Cause” components” or “Root Cause” or the so so called “culprit” components. Typical example, is dump disk driver symbolic references found in execution residue on the raw stack of a running bugchecking thread:

0: kd> !thread
THREAD fffffa8002bdebb0  Cid 03c4.03f0  Teb: 000007fffffde000 Win32Thread: fffff900c20f9810 RUNNING on processor 0
IRP List:
    fffffa8002b986f0: (0006,0118) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 fffff88005346920
Owning Process            fffffa80035bec10       Image:         Application.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      35246          Ticks: 7 (0:00:00:00.109)
Context Switch Count      1595                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.031
Win32 Start Address Application (0x0000000140002708)
Stack Init fffffa600495ddb0 Current fffffa600495d720
Base fffffa600495e000 Limit fffffa6004955000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Call Site
fffffa60`0495d558 fffff800`0186e3ee : nt!KeBugCheckEx
fffffa60`0495d560 fffff800`0186d2cb : nt!KiBugCheckDispatch+0×6e
fffffa60`0495d6a0 fffffa60`03d5917a : nt!KiPageFault+0×20b (TrapFrame @ fffffa60`0495d6a0)

0: kd> dps fffffa6004955000 fffffa600495e000
fffffa60`04955000  00d4d0c8`00d4d0c8
fffffa60`04955008  00d4d0c8`00d4d0c8
fffffa60`04955010  00d4d0c8`00d4d0c8
fffffa60`0495c7e0  00000000`00000001
fffffa60`0495c7e8  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c7f0  fffffa80`024c05a8
fffffa60`0495c7f8  fffffa60`02869ad4 dump_dumpata!IdeDumpNotification+0×1a4
fffffa60`0495c800  fffffa60`0495cb00
fffffa60`0495c808  fffff800`0182ff34 nt!output_l+0×6c0
fffffa60`0495c810  fffffa60`02860110 crashdmp!StrBeginningDump
fffffa60`0495c818  fffffa60`0495cb00
fffffa60`0495c820  00000000`00000000
fffffa60`0495c828  fffffa60`02869b18 dump_dumpata!IdeDumpNotification+0×1e8
fffffa60`0495c830  00000000`00000000
fffffa60`0495c838  fffffa60`0495c8c0
fffffa60`0495c840  00000000`00000000
fffffa60`0495c848  fffffa60`00000024
fffffa60`0495c850  00000000`ffffffff
fffffa60`0495c858  00000000`00000000
fffffa60`0495c860  00000000`00000000
fffffa60`0495c868  fffffa60`0495cb00
fffffa60`0495c870  fffffa80`00000000
fffffa60`0495c878  00000000`00000000
fffffa60`0495c880  00000000`00000101
fffffa60`0495c888  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c890  fffffa60`0495cb0f
fffffa60`0495c898  fffff800`0182ff34 nt!output_l+0×6c0
fffffa60`0495c8a0  fffffa60`0495cb0f
fffffa60`0495c8a8  fffffa60`0495cb90
fffffa60`0495c8b0  00000000`00000040
fffffa60`0495c8b8  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c8c0  fffffa80`024c0728
fffffa60`0495c8c8  fffffa80`024c0728
fffffa60`0495c8d0  00000001`00000000
fffffa60`0495c8d8  fffffa60`00000026
fffffa60`0495c8e0  00000000`ffffffff
fffffa60`0495c8e8  00000000`00000000
fffffa60`0495c8f0  fffffa80`00000000
fffffa60`0495c8f8  fffffa60`0495cb90
fffffa60`0495c900  00000000`00000000
fffffa60`0495c908  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c910  00000000`00000000
fffffa60`0495c918  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c920  fffff880`05311010
fffffa60`0495c928  00000000`00000002
fffffa60`0495c930  fffffa60`02875094 dump_SATA_Driver!AhciAdapterControl
fffffa60`0495c938  fffffa80`024c6018
fffffa60`0495c940  fffffa80`024c0728
fffffa60`0495c948  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c950  fffffa80`024c0728
fffffa60`0495c958  00000000`00000000
fffffa60`0495c960  fffffa60`0495ca18
fffffa60`0495c968  00000000`00000000
fffffa60`0495c970  fffffa80`024c0728
fffffa60`0495c978  fffffa60`02876427 dump_SATA_Driver!AhciHwInitialize+0×337
fffffa60`0495c980  fffffa80`024c0be6
fffffa60`0495c988  fffffa60`0286a459 dump_dumpata!IdeDumpWaitOnRequest+0×79
fffffa60`0495c990  00000000`00000000
fffffa60`0495c998  00000000`0000023a
fffffa60`0495c9a0  20474e55`534d4153
fffffa60`0495c9a8  204a4831`36314448
fffffa60`0495c9b0  20202020`20202020
fffffa60`0495c9b8  20202020`20202020
fffffa60`0495c9c0  fffffa80`024c05a8
fffffa60`0495c9c8  fffffa60`02869b18 dump_dumpata!IdeDumpNotification+0×1e8
fffffa60`0495c9d0  00000000`00000000
fffffa60`0495c9d8  fffffa60`0495ca60
fffffa60`0495c9e0  00000000`00000001
fffffa60`0495c9e8  fffffa60`02869396 dump_dumpata!IdeDumpMiniportChannelInitialize+0×236
fffffa60`0495c9f0  fffffa80`024c05a8
fffffa60`0495c9f8  fffffa60`02869ad4 dump_dumpata!IdeDumpNotification+0×1a4
fffffa60`0495ca00  00000000`00000000
fffffa60`0495ca08  fffffa60`0495ca90
fffffa60`0495ca10  00000000`00000001
fffffa60`0495ca18  00000001`00000038
fffffa60`0495ca20  00000000`10010000
fffffa60`0495ca28  00000000`00000003
fffffa60`0495ca30  fffffa80`024c05a8
fffffa60`0495ca38  fffffa60`0286a954 dump_dumpata!AtaPortGetPhysicalAddress+0×2c
fffffa60`0495ca40  fffffa80`024c0728
fffffa60`0495ca48  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495ca50  00000000`00000001
fffffa60`0495ca58  0000003f`022a8856
fffffa60`0495ca60  fffffa80`0000000c
fffffa60`0495ca68  fffffa80`024c0728
fffffa60`0495ca70  00000000`00000200
fffffa60`0495ca78  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495ca80  fffffa80`024c0728
fffffa60`0495ca88  ffff6226`4f5f3eb8
fffffa60`0495ca90  00000000`00000010
fffffa60`0495ca98  fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495caa0  fffffa80`024c05a8
fffffa60`0495caa8  fffffa60`02875a0d dump_SATA_Driver!AhciHwStartIo+0×69d
fffffa60`0495cab0  fffffa80`024c0728
fffffa60`0495cab8  00000000`00000000
fffffa60`0495cac0  00000000`00000001
fffffa60`0495cac8  fffff800`018f3dfc nt!DisplayCharacter+0×5c
fffffa60`0495cad0  00000000`00000000
fffffa60`0495cad8  fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495cae0  00000000`00010000
fffffa60`0495cae8  00000000`00000000
fffffa60`0495caf0  fffffa60`0495cd10
fffffa60`0495caf8  fffffa60`0495cc00
fffffa60`0495cb00  fffffa80`024c01c0
fffffa60`0495cb08  fffffa60`02875c3f dump_SATA_Driver!AhciHwInterrupt+0×2b
fffffa60`0495cb10  fffffa80`024c05a8
fffffa60`0495cb18  00000000`00000000
fffffa60`0495cb20  00000000`00000000
fffffa60`0495cb28  fffff800`01d406c9 hal!KeStallExecutionProcessor+0×25
fffffa60`0495cb30  00000000`00010000
fffffa60`0495cb38  00000000`00000000
fffffa60`0495cb40  fffffa60`0495cd10
fffffa60`0495cb48  fffffa60`0495cc00
fffffa60`0495cb50  00000000`00000000
fffffa60`0495cb58  fffffa60`0286a429 dump_dumpata!IdeDumpWaitOnRequest+0×49
fffffa60`0495cb60  fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cb68  00000000`d8bda325
fffffa60`0495cb70  00000000`00000000
fffffa60`0495cb78  00000000`0000033e
fffffa60`0495cb80  00000000`00000000
fffffa60`0495cb88  fffffa60`028694d2 dump_dumpata!IdeDumpWritePending+0xee
fffffa60`0495cb90  fffffa80`024c0000
fffffa60`0495cb98  fffffa80`024c01c0
fffffa60`0495cba0  00000000`00000000
fffffa60`0495cba8  00000000`00000000
fffffa60`0495cbb0  fffffa80`024c01c0
fffffa60`0495cbb8  fffffa80`01e3c740
fffffa60`0495cbc0  00000000`00010000
fffffa60`0495cbc8  00000000`00000000
fffffa60`0495cbd0  00000000`0c01f000
fffffa60`0495cbd8  fffffa60`0285bca9 crashdmp!WritePageSpanToDisk+0×181
fffffa60`0495cbe0  00000000`83d81000
fffffa60`0495cbe8  00000000`00000000
fffffa60`0495cbf0  fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cbf8  00000000`00000002
fffffa60`0495cc00  00000000`00000000
fffffa60`0495cc08  00000000`00030000
fffffa60`0495cc10  00000000`00000000
fffffa60`0495cc18  fffffa60`00441000
fffffa60`0495cc20  fffffa60`00441000
fffffa60`0495cc28  00000000`00010000
fffffa60`0495cc30  00000000`0000c080
fffffa60`0495cc38  00000000`0000c081
fffffa60`0495cc40  00000000`0000c082
fffffa60`0495cc48  00000000`0000c083
fffffa60`0495cc50  00000000`0000c084
fffffa60`0495cc58  00000000`0000c085
fffffa60`0495cc60  00000000`0000c086
fffffa60`0495cc68  00000000`0000c087
fffffa60`0495cc70  00000000`0000c088
fffffa60`0495cc78  00000000`0000c089
fffffa60`0495cc80  00000000`0000c08a
fffffa60`0495cc88  00000000`0000c08b
fffffa60`0495cc90  00000000`0000c08c
fffffa60`0495cc98  00000000`0000c08d
fffffa60`0495cca0  00000000`0000c08e
fffffa60`0495cca8  00000000`0000c08f
fffffa60`0495ccb0  00000000`00000000
fffffa60`0495ccb8  00000000`00000000
fffffa60`0495ccc0  00000000`00000000
fffffa60`0495ccc8  00000000`00000010
fffffa60`0495ccd0  00000000`0000c01d
fffffa60`0495ccd8  fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cce0  00000000`0000bf80
fffffa60`0495cce8  00000000`00000001
fffffa60`0495ccf0  00000000`00000000
fffffa60`0495ccf8  fffffa80`01e353d0
fffffa60`0495cd00  fffffa80`01e353f8
fffffa60`0495cd08  fffffa60`0285bacc crashdmp!WriteFullDump+0×70
fffffa60`0495cd10  00000002`3a3d8000
fffffa60`0495cd18  00000000`0000c080
fffffa60`0495cd20  fffffa80`00000000
fffffa60`0495cd28  fffffa60`0285c9c0 crashdmp!CrashdmpWriteRoutine
fffffa60`0495cd30  fffff880`05311010
fffffa60`0495cd38  00000000`00000002
fffffa60`0495cd40  fffffa60`0495cf70
fffffa60`0495cd48  00000000`00000000
fffffa60`0495cd50  fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cd58  fffffa60`0285b835 crashdmp!DumpWrite+0xc5
fffffa60`0495cd60  00000000`00000000
fffffa60`0495cd68  00000000`0000000f
fffffa60`0495cd70  00000000`00000001
fffffa60`0495cd78  fffffa60`00000001
fffffa60`0495cd80  fffffa80`02bdebb0
fffffa60`0495cd88  fffffa60`0285b153 crashdmp!CrashdmpWrite+0×57
fffffa60`0495cd90  00000000`00000000
fffffa60`0495cd98  fffffa60`028602f0 crashdmp!StrInitPortDriver
fffffa60`0495cda0  00000000`00000000
fffffa60`0495cda8  fffffa60`02860a00 crashdmp!ContextCopy
fffffa60`0495cdb0  00000000`00000000
fffffa60`0495cdb8  fffff800`01902764 nt!IoWriteCrashDump+0×3f4
fffffa60`0495cdc0  fffffa60`0495ce00
fffffa60`0495cdc8  00000028`00000025
fffffa60`0495cdd0  fffff800`018afd40 nt! ?? ::FNODOBFM::`string’
fffffa60`0495cdd8  00000000`000000d1
fffffa60`0495cde0  fffff880`05311010
fffffa60`0495cde8  00000000`00000002
fffffa60`0495cdf0  00000000`00000000
fffffa60`0495cdf8  fffffa60`03d5917a
fffffa60`0495ce00  202a2a2a`0a0d0a0d
fffffa60`0495ce08  7830203a`504f5453
fffffa60`0495ce10  31443030`30303030
fffffa60`0495ce18  46464646`78302820
fffffa60`0495ce20  31333530`30383846
fffffa60`0495ce28  fffff800`018f5f83 nt!VidDisplayString+0×143
fffffa60`0495ce30  30303030`30300030
fffffa60`0495ce38  2c323030`30303030
fffffa60`0495ce40  30303030`30307830
fffffa60`0495ce48  30303030`30303030
fffffa60`0495ce50  46464678`302c3030
fffffa60`0495ce58  fffff800`018fe040 nt!KiInvokeBugCheckEntryCallbacks+0×80
fffffa60`0495ce60  fffffa80`02bdebb0
fffffa60`0495ce68  fffff800`01921d52 nt!InbvDisplayString+0×72
fffffa60`0495ce70  fffff880`05311000
fffffa60`0495ce78  fffff800`01d406c9 hal!KeStallExecutionProcessor+0×25
fffffa60`0495ce80  00000000`00000001
fffffa60`0495ce88  00000000`0000000a
fffffa60`0495ce90  fffffa60`03d5917a
fffffa60`0495ce98  00000000`40000082
fffffa60`0495cea0  00000000`00000001
fffffa60`0495cea8  fffff800`01922c3e nt!KeBugCheck2+0×92e
fffffa60`0495ceb0  fffff800`000000d1
fffffa60`0495ceb8  00000000`000004d0
fffffa60`0495cec0  fffff800`01a43640 nt!KiProcessorBlock
fffffa60`0495cec8  00000000`0000000a
fffffa60`0495ced0  fffffa60`03d5917a
fffffa60`0495ced8  fffffa60`0495cf70
fffffa60`0495cee0  fffffa80`02bdebb0
fffffa60`0495cee8  00000000`00000000
fffffa60`0495cef0  00000000`00000000
fffffa60`0495cef8  fffffa80`02bdebb0
fffffa60`0495cf00  00000000`c21a6d00
fffffa60`0495cf08  00000000`00000000
fffffa60`0495cf10  fffff800`0198e7a0 nt!KiInitialPCR+0×2a0
fffffa60`0495cf18  fffff800`0198e680 nt!KiInitialPCR+0×180
fffffa60`0495cf20  fffffa80`02bb7320
fffffa60`0495cf28  00000000`00000000
fffffa60`0495cf30  00000000`00000000
fffffa60`0495cf38  fffff960`00000003
fffffa60`0495cf40  fffffa60`0495e000
fffffa60`0495cf48  fffffa60`04955000
fffffa60`0495cf50  00000001`c0643000
fffffa60`0495cf58  00000000`00000000
fffffa60`0495cf60  fffff900`c06ca53c
fffffa60`0495cf68  fffffa60`0495d090
fffffa60`0495cf70  00000000`00000000
fffffa60`0495cf78  00000000`00000000
fffffa60`0495cf80  00000000`00000000
fffffa60`0495cf88  00000000`00000000
fffffa60`0495cf90  00000000`00000000
fffffa60`0495cf98  00000000`00000000
fffffa60`0495cfa0  00001f80`0010000f
fffffa60`0495cfa8  0053002b`002b0010
fffffa60`0495cfb0  00000286`0018002b
fffffa60`0495cfb8  00000000`00000000
fffffa60`0495cfc0  00000000`00000000
fffffa60`0495cfc8  00000000`00000000
fffffa60`0495cfd0  00000000`00000000
fffffa60`0495cfd8  00000000`00000000
fffffa60`0495cfe0  00000000`00000000
fffffa60`0495cfe8  fffffa60`0495d660
fffffa60`0495cff0  00000000`0000000a
fffffa60`0495cff8  fffff880`05311010
fffffa60`0495d000  fffff880`05311010
fffffa60`0495d008  fffffa60`0495d558
fffffa60`0495d010  fffffa60`0495d720
fffffa60`0495d018  fffffa80`02b986f0
fffffa60`0495d020  fffffa80`02b98720
fffffa60`0495d028  00000000`00000002
fffffa60`0495d030  00000000`00000000
fffffa60`0495d038  fffffa60`03d5917a
fffffa60`0495d040  00000000`000001f1
fffffa60`0495d048  fffffa80`026a9df0
fffffa60`0495d050  00000000`00000001
fffffa60`0495d058  00000000`83360018
fffffa60`0495d060  fffffa80`02b3ee40
fffffa60`0495d068  fffff800`0186e650 nt!KeBugCheckEx
fffffa60`0495d070  00000000`00000000
fffffa60`0495d078  00000000`00000000
fffffa60`0495d080  00000000`00000000
fffffa60`0495d088  00000000`00000000
fffffa60`0495d090  00000000`00000000
fffffa60`0495d098  00000000`00000000
fffffa60`0495d0a0  00000000`00000000

If a BSOD was reported after installing new drivers we shouldn’t suspect SATA_Driver package here because its components would almost always be present on any bugcheck thread as referenced after a bugcheck cause. There presence is the “effect”. This example might seem trivial and pointless but I’ve seen some memory dump analysis conclusions based on the reversal of causes and effects.

- Dmitry Vostokov @ -

Software Victimology (Part 1)

Friday, October 23rd, 2009

What prompted me to found this discipline (that is supposed to be a sister discipline of software criminology, software security, secure code construction and software defect construction) is understanding that some software components are innocent victims of other component coding mistakes or deliberate subversion and some start as a part of crimeware but eventually become victims themselves (they crash, hang, spike, leak, are dumped, subverted, etc.). I would also like to borrow and reuse the neglected term victimware here in a broad sense. More on this later as I have to switch to software trace analysis patterns.

- Dmitry Vostokov @ -

There Ought to be a Planet at that Location!

Thursday, October 22nd, 2009

One ETW trace pointed to a set of intermittent symptoms (messages were simplified for this post):

#        PID        TID        Message 
31278    2300       7060       RequestXMLData entry
31281    2300       7060       RequestXMLData: XML error     

Searching for issues having this error only pointed to a case with a mixed software product environment where some servers had the product version X and other servers the product version X+1. However, in the new case the customer claimed that he had only the product version X+1 on all production servers. We insisted and, after the closer inspection, servers with the product X were found… 

- Dmitry Vostokov @ -

DebugWare Patterns (Part 11)

Wednesday, October 21st, 2009

The domain of some troubleshooting and debugging tools sometimes spans several computers, for example, a client, a server, and a middleware server (or a frontware server like a web server). Because such tools are small, there is no need to devise yet another distributed architecture but force the tool to differentiate its functionality based on a command line parameters. For example, the following tool implements Checklist pattern and chooses the right checklist from a checklist repository based on a command line option:

c:\> GreatTool  -client

c:\> GreatTool  -server

This nicely fits with a case when a server is also a client to another server and we need to troubleshoot its client and server functionality on the same physical computer. 

- Dmitry Vostokov @ -

DebugWare Patterns (Part 10)

Tuesday, October 20th, 2009

Many troubleshooting tools and their components (like TUW modules from checklists) reuse the same code over and over, for example, querying registry keys or environment variables. It is beneficial to have a package of supporting modules (or a library) for code reuse purposes, hence the name of this pattern, Supporting Module (SM). In the context of Checklist pattern we can refine its UML component diagram to highlight that TUWs are using some shared SM components:

- Dmitry Vostokov @ -

Notes on Memoidealism (1.6)

Tuesday, October 20th, 2009

Although everything changes in the philosophy of Heraclitus, its Urstoff, fire, is eternal, indestructable. The same can be said about Memory and its memory fragments. Memoidealism has also some proximity with the notion of everlasting existence of thought of or spoken named objects in the philosophy of Parmenides. Once we think of or speak of something we immediately recall a memory fragment from and, at the same time, commit it to Memory again, perhaps as a different memory fragment, an assembled new memory trace.

- Dmitry Vostokov @ -

Bugtation No.106

Monday, October 19th, 2009

Computation “is a brief gasp between one” crash dump “and another.”

Ezra Weston Loomis Pound, Subbugtated from Poetry: The Basics (Chapter 3), Jeffrey Wainwright

- Dmitry Vostokov @ -

Software Maintenance Institute

Friday, October 16th, 2009

Memory Analysis and Debugging Institute (MA&DI), DA+TA Portal ( + and OpenTask establish

Software Maintenance Institute (SMI)

More details about R&D Institute of Software Maintenance will be announced soon.

- Dmitry Vostokov @ -

Subversion of Pause (CoTweet Annoyances)

Thursday, October 15th, 2009

Noticed (twice) that unattended IE 8 with CoTweet page opened spikes 50% CPU on my 2 processor PC (distorting music I listen to) despite the very big image of Pause button across the entire IE tab screen. I saved the dump and it shows the longest HTML processing user space thread stack I have ever seen:

0:000> !runaway
 User Mode Time
  Thread       Time
  17:f18       0 days 14:13:41.290
  23:6b0       0 days 0:00:41.309
   4:c88       0 days 0:00:29.031
  10:129c      0 days 0:00:03.853
   3:126c      0 days 0:00:02.620
  16:f98       0 days 0:00:01.934
  18:f8c       0 days 0:00:00.702
   2:cd4       0 days 0:00:00.280
   8:c40       0 days 0:00:00.109
   7:1348      0 days 0:00:00.093
  11:ee4       0 days 0:00:00.015
  24:490       0 days 0:00:00.000
  22:d98       0 days 0:00:00.000
  21:104c      0 days 0:00:00.000
  20:1090      0 days 0:00:00.000
  19:f60       0 days 0:00:00.000
  15:1150      0 days 0:00:00.000
  14:1048      0 days 0:00:00.000
  13:d28       0 days 0:00:00.000
  12:c68       0 days 0:00:00.000
   9:ab0       0 days 0:00:00.000
   6:69c       0 days 0:00:00.000
   5:c28       0 days 0:00:00.000
   1:1264      0 days 0:00:00.000
   0:fa8       0 days 0:00:00.000

0:017:x86> kL 100
ChildEBP RetAddr 
0c3b6f48 70072d55 mshtml!CPtsClient::CreateParaclient+0x9
0c3b6f9c 70072288 mshtml!Ptls5::FsConductCensusZooCore+0x30a
0c3b7094 70071720 mshtml!Ptls5::FsConductCensusZooCore+0x62f
0c3b70d4 70072647 mshtml!Ptls5::FsConductCensusZooCore+0x9e1
0c3b71a4 700ce813 mshtml!Ptls5::FsFormatZooCore+0x2f6
0c3b71d8 700cc65b mshtml!Ptls5::FsFormatZoo+0x51
0c3b730c 700cc147 mshtml!Ptls5::FsCreateSubpageCore+0x6f7
0c3b733c 700ccb58 mshtml!Ptls5::FsCreateSubpage+0x82
0c3b73fc 700c9640 mshtml!Ptls5::FsGetSavedPel+0x6ce
0c3b751c 700cae11 mshtml!Ptls5::FsFormatPelCore+0x73d
0c3b75f0 700cb347 mshtml!Ptls5::FsAdjustPageVertical+0x58e
0c3b7690 700c9e1d mshtml!Ptls5::FsAdjustPageVertical+0x15cb
0c3b76e4 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3b778c 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3b7860 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3b7974 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3b79c0 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3b7b20 700cfc68 mshtml!Ptls5::FsFillTrack+0x2ec
0c3b7b94 700cf2a3 mshtml!Ptls5::FsFillTrackWraper+0xda
0c3b7c50 700cf1cd mshtml!Ptls5::FsConductCensusSpanLayout+0x1b6
0c3b7cac 700cf12c mshtml!Ptls5::FsConductCensusSpanLayout+0x41c
0c3b7ce4 700cc416 mshtml!Ptls5::FsFillLayoutWithSpanAreas+0x219
0c3b7e00 700cc147 mshtml!Ptls5::FsCreateSubpageCore+0x45c
0c3b7e30 700ccb58 mshtml!Ptls5::FsCreateSubpage+0x82
0c3b7ef0 700c9640 mshtml!Ptls5::FsGetSavedPel+0x6ce
0c3b8010 700cae11 mshtml!Ptls5::FsFormatPelCore+0x73d
0c3b80e4 700cb347 mshtml!Ptls5::FsAdjustPageVertical+0x58e
0c3b8184 700c9e1d mshtml!Ptls5::FsAdjustPageVertical+0x15cb
0c3b81d8 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3b8280 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3b8354 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3b8468 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3b84b4 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3b8614 700d0fc0 mshtml!Ptls5::FsFillTrack+0x2ec
0c3b8694 700c9e1d mshtml!Ptls5::FsFormatParaTrelPost+0x841
0c3b86e8 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3b8790 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3b8864 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3b8978 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3b89c4 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3b8b24 700d0fc0 mshtml!Ptls5::FsFillTrack+0x2ec
0c3b8ba4 700c9e1d mshtml!Ptls5::FsFormatParaTrelPost+0x841
0c3b8bf8 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3b8ca0 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3b8d74 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3b8e88 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3b8ed4 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3b9034 700d0fc0 mshtml!Ptls5::FsFillTrack+0x2ec
0c3b90b4 700c9e1d mshtml!Ptls5::FsFormatParaTrelPost+0x841
0c3b9108 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3b91b0 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3b9284 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3b9398 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3b93e4 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3b9544 700d0fc0 mshtml!Ptls5::FsFillTrack+0x2ec
0c3b95c4 700c9e1d mshtml!Ptls5::FsFormatParaTrelPost+0x841
0c3b9618 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3b96c0 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3b9794 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3b98a8 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3b98f4 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3b9a54 700cfc68 mshtml!Ptls5::FsFillTrack+0x2ec
0c3b9ac8 700cf2a3 mshtml!Ptls5::FsFillTrackWraper+0xda
0c3b9b84 700cf1cd mshtml!Ptls5::FsConductCensusSpanLayout+0x1b6
0c3b9be0 700cf12c mshtml!Ptls5::FsConductCensusSpanLayout+0x41c
0c3b9c18 700cc416 mshtml!Ptls5::FsFillLayoutWithSpanAreas+0x219
0c3b9d34 700cc147 mshtml!Ptls5::FsCreateSubpageCore+0x45c
0c3b9d64 700ccb58 mshtml!Ptls5::FsCreateSubpage+0x82
0c3b9e24 700c9640 mshtml!Ptls5::FsGetSavedPel+0x6ce
0c3b9f44 700cae11 mshtml!Ptls5::FsFormatPelCore+0x73d
0c3ba018 700cb347 mshtml!Ptls5::FsAdjustPageVertical+0x58e
0c3ba0b8 700c9e1d mshtml!Ptls5::FsAdjustPageVertical+0x15cb
0c3ba10c 700cd0e1 mshtml!Ptls5::FsDuplicateParaFormatResult+0x2be
0c3ba1b4 700cf796 mshtml!Ptls5::FsFormatOrReusePara+0x400
0c3ba288 700cec9c mshtml!Ptls5::FsDestroyBrtrackElements+0x25d
0c3ba39c 700ceaa0 mshtml!Ptls5::FsFormatDelayedParas+0x1b2d
0c3ba3e8 700c7e87 mshtml!Ptls5::FsFillTrackMainFlow+0x40
0c3ba548 700cfc68 mshtml!Ptls5::FsFillTrack+0x2ec
0c3ba5bc 700cf2a3 mshtml!Ptls5::FsFillTrackWraper+0xda
0c3ba678 700d55b3 mshtml!Ptls5::FsConductCensusSpanLayout+0x1b6
0c3ba760 700d6aac mshtml!Ptls5::FsFormatMathParaInGivenRectangleFiniteMain+0x1232
0c3ba794 700d7957 mshtml!Ptls5::FsFormatMathParaInGivenRectangleFiniteMain+0x148b
0c3ba7dc 700d7807 mshtml!Ptls5::FsFillMultiColumnLayout+0xab
0c3ba8bc 700d7770 mshtml!Ptls5::FsDestroySpanLayoutContent+0x160
0c3ba904 700d758b mshtml!Ptls5::FsFillLayoutWithSpanAreas+0x213
0c3ba950 700d6367 mshtml!Ptls5::FsConductCensusGeneralSection+0x152
0c3baac4 700d6208 mshtml!Ptls5::FsFormatGeneralSection+0xee
0c3bab30 700d6078 mshtml!Ptls5::FsFormatSection+0x111
0c3bac10 700d6691 mshtml!Ptls5::FsDestroyPageBodyBreakRecord+0x279
0c3bad54 700d6f17 mshtml!Ptls5::FsFormatPageBody+0x1d7
0c3bae8c 700f595e mshtml!Ptls5::FsCreatePageBottomlessCore+0x262
0c3baeb0 700f590b mshtml!Ptls5::FsUpdateBottomlessPage+0x71
0c3baf54 700cba8e mshtml!CCssDocumentLayout::GetPage+0x54d
0c3bb0c4 7000bf07 mshtml!CCssPageLayout::CalcSizeVirtual+0x247
0c3bb1fc 7001eb36 mshtml!CLayout::CalcSize+0x2b8
0c3bb2f8 701ea7a9 mshtml!CLayout::DoLayout+0x11d
0c3bb30c 6fff932b mshtml!CCssPageLayout::Notify+0x140
0c3bb318 6fff741e mshtml!NotifyElement+0x41
0c3bb36c 6fff73b5 mshtml!CMarkup::SendNotification+0x60
0c3bb394 7000248a mshtml!CMarkup::Notify+0xd4
0c3bb3dc 6ffd54a2 mshtml!CElement::SendNotification+0x4a
0c3bb400 6fff08c3 mshtml!CElement::EnsureRecalcNotify+0x15f
0c3bb43c 6fff0865 mshtml!CElement::get_clientWidth_Logical+0x54
0c3bb450 7013257c mshtml!CElement::get_clientWidth+0x27
0c3bb470 7003afc3 mshtml!G_long+0x7b
0c3bb4e4 70047001 mshtml!CBase::ContextInvokeEx+0x5d1
0c3bb534 70052da4 mshtml!CElement::ContextInvokeEx+0x9d
0c3bb560 7003a9c1 mshtml!CInput::VersionedInvokeEx+0x2d
0c3bb5b0 7473389a mshtml!PlainInvokeEx+0xea
0c3bb5f0 747337e6 jscript!IDispatchExInvokeEx2+0xf8
0c3bb62c 74734d26 jscript!IDispatchExInvokeEx+0x6a
0c3bb6ec 74734c80 jscript!InvokeDispatchEx+0x98
0c3bb720 7474aadb jscript!VAR::InvokeByName+0x135
0c3bb7d4 74729679 jscript!VAR::InvokeByVar+0x78
0c3bb968 747311ab jscript!CScriptRuntime::Run+0x446
0c3bba50 747310e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
0c3bba9c 74732805 jscript!ScrFncObj::Call+0x8f
0c3bbb20 747326c5 jscript!NameTbl::InvokeInternal+0x2a2
0c3bbb54 747341fc jscript!VAR::InvokeByDispID+0x17c
0c3bbb94 747322c1 jscript!VAR::InvokeJSObj<SYM *>+0xb8
0c3bbbd0 74732b6d jscript!VAR::InvokeByName+0x170
0c3bbc1c 74734035 jscript!VAR::InvokeDispName+0x7a
0c3bbc4c 74732f14 jscript!VAR::InvokeByDispID+0xce
0c3bbde8 747311ab jscript!CScriptRuntime::Run+0x29e0
0c3bbed0 747310e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
0c3bbf1c 74732805 jscript!ScrFncObj::Call+0x8f
0c3bbfa0 747326c5 jscript!NameTbl::InvokeInternal+0x2a2
0c3bbfd4 747341fc jscript!VAR::InvokeByDispID+0x17c
0c3bc014 747322c1 jscript!VAR::InvokeJSObj<SYM *>+0xb8
0c3bc050 74732b6d jscript!VAR::InvokeByName+0x170
0c3bc09c 74734035 jscript!VAR::InvokeDispName+0x7a
0c3bc0cc 74734d93 jscript!VAR::InvokeByDispID+0xce
0c3bc268 747311ab jscript!CScriptRuntime::Run+0x2abe
0c3bc350 747310e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
0c3bc39c 74732805 jscript!ScrFncObj::Call+0x8f
0c3bc420 747326c5 jscript!NameTbl::InvokeInternal+0x2a2
0c3bc454 7474587a jscript!VAR::InvokeByDispID+0x17c
0c3bc514 74734127 jscript!JsFncApply+0x2ba
0c3bc57c 74732805 jscript!NatFncObj::Call+0x103
0c3bc600 747326c5 jscript!NameTbl::InvokeInternal+0x2a2
0c3bc634 747341fc jscript!VAR::InvokeByDispID+0x17c
0c3bc674 747322c1 jscript!VAR::InvokeJSObj<SYM *>+0xb8
0c3bc6b0 74732b6d jscript!VAR::InvokeByName+0x170
0c3bc6fc 74734035 jscript!VAR::InvokeDispName+0x7a
0c3bc72c 74732f14 jscript!VAR::InvokeByDispID+0xce
0c3bc8c8 747311ab jscript!CScriptRuntime::Run+0x29e0
0c3bc9b0 747310e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
0c3bc9fc 74732805 jscript!ScrFncObj::Call+0x8f
0c3bca80 747326c5 jscript!NameTbl::InvokeInternal+0x2a2
0c3bcab4 7474587a jscript!VAR::InvokeByDispID+0x17c
0c3bcb74 74734127 jscript!JsFncApply+0x2ba
0c3bcbdc 74732805 jscript!NatFncObj::Call+0x103
0c3bcc60 747326c5 jscript!NameTbl::InvokeInternal+0x2a2
0c3bcc94 747341fc jscript!VAR::InvokeByDispID+0x17c
0c3bccd4 747322c1 jscript!VAR::InvokeJSObj<SYM *>+0xb8
0c3bcd10 74732b6d jscript!VAR::InvokeByName+0x170
0c3bcd5c 74734035 jscript!VAR::InvokeDispName+0x7a
0c3bcd88 74732f14 jscript!VAR::InvokeByDispID+0xce
0c3bcf24 747311ab jscript!CScriptRuntime::Run+0x29e0
0c3bd00c 747310e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
0c3bd058 74730f13 jscript!ScrFncObj::Call+0x8f
0c3bd0d4 7473365e jscript!CSession::Execute+0x175
0c3bd1bc 747334ea jscript!NameTbl::InvokeDef+0x1b8
0c3bd240 6ffb1e59 jscript!NameTbl::InvokeEx+0x129
0c3bd29c 6ffb1da0 mshtml!CBase::InvokeDispatchExtraParam+0x139
0c3bd35c 6ff93328 mshtml!CBase::InvokeAttachEvents+0x160
0c3bd4ac 6ffad74e mshtml!COmWindowProxy::FireEvent+0x1ef
0c3bd4d0 6ffa8553 mshtml!COmWindowProxy::Fire_onresize+0x20
0c3bd4d8 6ffa84df mshtml!CElement::Fire_onresize+0x54
0c3bd508 7002e6d1 mshtml!CView::ExecuteEventTasks+0x21c
0c3bd54c 70047f49 mshtml!CView::EnsureView+0x325
0c3bd9f0 7002eeb9 mshtml!CDoc::OnPaint+0x147
0c3bda24 70059527 mshtml!CServer::OnWindowMessage+0x38f
0c3bdb4c 70059401 mshtml!CDoc::OnWindowMessage+0x16c
0c3bdb78 75b78817 mshtml!CServer::WndProc+0x78
0c3bdba4 75b7898e user32!InternalCallWinProc+0x23
0c3bdc1c 75b79d14 user32!UserCallWinProcCheckWow+0x109
0c3bdc78 75b79d85 user32!DispatchClientMessage+0xe0
0c3bdcb4 7740e476 user32!__fnDWORD+0x2b
0c3bdd40 7062ba5d ntdll_773f0000!KiUserCallbackDispatcher+0x2e
0c3bfe48 705d30d6 ieframe!CTabWindow::_TabWindowThreadProc+0x54c
0c3bff00 7566408d ieframe!LCIETab_ThreadProc+0x2c1
0c3bff10 7593eccb iertutil!CIsoScope::RegisterThread+0xab
0c3bff1c 7746d24d kernel32!BaseThreadInitThunk+0xe
0c3bff5c 7746d45f ntdll_773f0000!__RtlUserThreadStart+0x23
0c3bff74 00000000 ntdll_773f0000!_RtlUserThreadStart+0x1b

- Dmitry Vostokov @ -

Essential and Derived Properties

Thursday, October 15th, 2009

Sometimes when we point to an activity or a problem in software traces and memory dumps, for example, “not enough space” message, we are asked to dig for certain properties, for example, a file size. Fortunately, some some of them are essential and has to be stored somewhere in various structures, for example:

0: kd> !handle
02b0: Object: 86f556f8  GrantedAccess: 0013019f Entry: d6e99560
Object: 86f556f8  Type: (89bba730) File
    ObjectHeader: 86f556e0 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \Temp\Tmp2523.tmp {HarddiskVolume1}

0: kd> !fileobj 86f556f8


Device Object: 0x89b36b18   \Driver\Ftdisk
Vpb: 0x89b55238
Event signalled
Access: Read Write Delete

Flags:  0x1cb042
       Synchronous IO
       Cache Supported
       Size Changed
       Temporary File
       Handle Created
       Fast IO Read
       Random Access

FsContext: 0xd6e25da0      FsContext2: 0xd6e25ee8
Private Cache Map: 0x88d44bb8
CurrentByteOffset: 45000
Cache Data:
  Section Object Pointers: 86f518dc
  Shared Cache Map: 86eafca0         File Offset: 45000 in VACB number 1
  Vacb: 89b83f08
  Your data is at: c33c5000

0: kd> dt _FILE_OBJECT 86f556f8
   +0×000 Type             : 5
   +0×002 Size             : 112
   +0×004 DeviceObject     : 0×89b36b18 _DEVICE_OBJECT
   +0×008 Vpb              : 0×89b55238 _VPB
   +0×00c FsContext        : 0xd6e25da0
   +0×010 FsContext2       : 0xd6e25ee8
   +0×014 SectionObjectPointer : 0×86f518dc _SECTION_OBJECT_POINTERS
   +0×018 PrivateCacheMap  : 0×88d44bb8
   +0×01c FinalStatus      : 0
   +0×020 RelatedFileObject : (null)
   +0×024 LockOperation    : 0 ”
   +0×025 DeletePending    : 0 ”
   +0×026 ReadAccess       : 0×1 ”
   +0×027 WriteAccess      : 0×1 ”
   +0×028 DeleteAccess     : 0×1 ”
   +0×029 SharedRead       : 0 ”
   +0×02a SharedWrite      : 0 ”
   +0×02b SharedDelete     : 0 ”
   +0×02c Flags            : 0×1cb042
   +0×030 FileName         : _UNICODE_STRING “\Temp\Tmp2523.tmp”
   +0×038 CurrentByteOffset : _LARGE_INTEGER 0×45000
   +0×040 Waiters          : 0
   +0×044 Busy             : 0
   +0×048 LastLock         : (null)
   +0×04c Lock             : _KEVENT
   +0×05c Event            : _KEVENT
   +0×06c CompletionContext : (null)

0: kd> dt _SECTION_OBJECT_POINTERS 0×86f518dc
   +0×000 DataSectionObject : 0×87056a58
   +0×004 SharedCacheMap   : 0×86eafca0
   +0×008 ImageSectionObject : (null)

0: kd> dt _SHARED_CACHE_MAP 0×86eafca0
   +0×000 NodeTypeCode     : 767
   +0×002 NodeByteSize     : 312
   +0×004 OpenCount        : 2
   +0×008 FileSize         : _LARGE_INTEGER 0×67000
   +0×010 BcbList          : _LIST_ENTRY [ 0×86eafcb0 - 0×86eafcb0 ]
   +0×018 SectionSize      : _LARGE_INTEGER 0×100000
   +0×020 ValidDataLength  : _LARGE_INTEGER 0×67000
   +0×028 ValidDataGoal    : _LARGE_INTEGER 0×67000
   +0×030 InitialVacbs     : [4] 0×89b802c0 _VACB
   +0×040 Vacbs            : 0×86eafcd0  -> 0×89b802c0 _VACB
   +0×044 FileObject       : 0×87095d68 _FILE_OBJECT
   +0×048 ActiveVacb       : 0×89b83f08 _VACB
   +0×04c NeedToZero       : (null)
   +0×050 ActivePage       : 0×40
   +0×054 NeedToZeroPage   : 0
   +0×058 ActiveVacbSpinLock : 0
   +0×05c VacbActiveCount  : 1
   +0×060 DirtyPages       : 0×22
   +0×064 SharedCacheMapLinks : _LIST_ENTRY [ 0×86c8f06c - 0×808acc20 ]
   +0×06c Flags            : 0×1400
   +0×070 Status           : 0
   +0×074 Mbcb             : 0×87f1a348 _MBCB
   +0×078 Section          : 0xd6653930
   +0×07c CreateEvent      : (null)
   +0×080 WaitOnActiveCount : (null)
   +0×084 PagesToWrite     : 0
   +0×088 BeyondLastFlush  : 0
   +0×090 Callbacks        : 0xf719664c _CACHE_MANAGER_CALLBACKS
   +0×094 LazyWriteContext : 0xd6e25da0
   +0×098 PrivateList      : _LIST_ENTRY [ 0×86eafdc4 - 0×88d44c04 ]
   +0×0a0 LogHandle        : (null)
   +0×0a4 FlushToLsnRoutine : (null)
   +0×0a8 DirtyPageThreshold : 0
   +0×0ac LazyWritePassCount : 0×3d7d1b
   +0×0b0 UninitializeEvent : (null)
   +0×0b4 NeedToZeroVacb   : (null)
   +0×0b8 BcbSpinLock      : 0
   +0×0bc Reserved         : (null)
   +0×0c0 Event            : _KEVENT
   +0×0d0 VacbPushLock     : _EX_PUSH_LOCK
   +0×0d8 PrivateCacheMap  : _PRIVATE_CACHE_MAP
   +0×130 WriteBehindWorkQueueEntry : (null)

An example of a derived property could be the difference between the time of logon (winlogon.exe start time) and the start of the currently running instance of a screen saver process.

- Dmitry Vostokov @ -

Hot Issues from Physicalist Artist Perspective

Thursday, October 15th, 2009

Materials: A3 paper, a bug, hot chilli sauce

- Dmitry Vostokov @ -

Dictionary of Debugging: Thread

Wednesday, October 14th, 2009


A mapping T: t -> IP, where t is a discreet time arrow and IP is a memory space (Instruction Pointer). Threads can also be viewed as braided strings in an abstract computational space. Another computer science definition centers on considering each individual thread as a Turing machine. One of popular definitions is a parallel activity inside a process. The following diagram depicts a thread in equi-bipartitional virtual memory space comprised from user and kernel spaces:

Synonyms: lightweight process


Also: virtual memory, memory space, user space, kernel space, breakpoint, process, fiber.

- Dmitry Vostokov @ -

6 years at Citrix!

Tuesday, October 13th, 2009

Threads in my process run very fast. Not long ago the watch variable had the value 5 and now it is 6:

switch (years_at_citrix)
  case 5:
    write_blog_post(”I’ve just passed 5 year mark … “);
    write_blog_post(”Shortly after celebrating 5 years … “);
  case 6:
    write_blog_post(”Threads in my process run very fast. Not long ago … “);
  case 7:
    // … TBD 

- Dmitry Vostokov @ -