This weekend noticed that it is in stock and this means that I soon get it in the post:
Advanced .NET Debugging (Addison-Wesley Microsoft Technology Series)
I plan to start reading it as soon as it arrives and put notes on my Software Generalist blog.
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes we can observe rare events when abnormal conditions that usually result in a system crash result in a milder problem, for example, a service is unavailable and not affecting other services and users. It was reported that an application was freezing during user session logoff. A complete memory dump was saved at that time and its stack trace collection (!stacks command) shows the following suspicious thread in a user process (all other threads were normally waiting):
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[89cfa960 Application.exe]
ea0.001c4c 89a11db0 0499cd1 Blocked DriverA+0×69db
[…]
0: kd> !thread 89a11db0 16
THREAD 89a11db0 Cid 0ea0.1c4c Teb: 7ffdf000 Win32Thread: bc347a48 WAIT: (Unknown) KernelMode Non-Alertable
89b87770 Unknown
b97004ac NotificationEvent
IRP List:
899e2668: (0006,0244) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap daf62b28
Owning Process 89cfa960 Image: Application.exe
Attached Process N/A Image: N/A
Wait Start TickCount 909331 Ticks: 4824273 (0:20:56:19.265)
Context Switch Count 186 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.093
*** ERROR: Module load completed but symbols could not be loaded for Application.exe
Win32 Start Address Application (0×00406b2a)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init b60ceb30 Current b60cdf10 Base b60cf000 Limit b60cb000 Call b60ceb34
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr Args to Child
b60cdf28 80833485 89a11db0 00000002 00000000 nt!KiSwapContext+0×26
b60cdf54 808294b9 dc399008 89b87748 b60ce01c nt!KiSwapThread+0×2e5
b60cdf88 b96d69db 00000002 b60cdfbc 00000001 nt!KeWaitForMultipleObjects+0×3d7
WARNING: Stack unwind information not available. Following frames may be wrong.
b60cdfe8 b96d719e 89b87748 dc399008 b60ce01c DriverA+0×69db
[…]
We notice “89b87770 Unknown” and double check what object the thread is waiting for:
0: kd> dp b60cdfbc L00000002
b60cdfbc 89b87770 b97004ac
These are exactly the same objects that are listed in !thread command output. We see that the second one is normal and resides in a nonpaged area:
0: kd> dt _DISPATCHER_HEADER b97004ac
ntdll!_DISPATCHER_HEADER
+0x000 Type : 0 ''
+0x001 Absolute : 0 ''
+0x001 NpxIrql : 0 ''
+0x002 Size : 0x4 ''
+0x002 Hand : 0x4 ''
+0x003 Inserted : 0 ''
+0x003 DebugActive : 0 ''
+0x000 Lock : 262144
+0x004 SignalState : 0
+0x008 WaitListHead : _LIST_ENTRY [ 0x89a11e70 - 0x89a11e70 ]
0: kd> !address b97004ac
a71e3000 - 13e1d000
Usage KernelSpaceUsageNonPagedSystem
The other looks like an invalid Random Object from the free nonpaged pool entry (it even says about itself that it is bad) that used to belong in the past to Configuration Manager:
0: kd> !pool 89b87770
Pool page 89b87770 region is Nonpaged pool
[...]
89b87540 size: 98 previous size: 40 (Allocated) File (Protected)
*89b875d8 size: 260 previous size: 98 (Free) *CMpa
Pooltag CMpa : registry post apcs, Binary : nt!cm
89b87838 size: 28 previous size: 260 (Allocated) FSfm
[…]
0: kd> dd 89b87770
89b87770 bad0b0b0 00000000 00000000 00000000
89b87780 8a04be01 00000000 89b87788 89b87788
89b87790 00150006 e56c6946 8993e208 89ab96b8
89b877a0 00000000 00000000 bad0b0b0 c0000800
89b877b0 02110004 63426343 88ebbf80 00001000
89b877c0 00199000 00000000 8993e238 88d0d248
89b877d0 0019a000 00000000 00000000 00000000
89b877e0 00000000 00000000 00000000 00000000
0: kd> dt _DISPATCHER_HEADER 89b87770
ntdll!_DISPATCHER_HEADER
+0×000 Type : 0xb0 ”
+0×001 Absolute : 0xb0 ”
+0×001 NpxIrql : 0xb0 ”
+0×002 Size : 0xd0 ”
+0×002 Hand : 0xd0 ”
+0×003 Inserted : 0xba ”
+0×003 DebugActive : 0xba ”
+0×000 Lock : -1160728400
+0×004 SignalState : 0
+0×008 WaitListHead : _LIST_ENTRY [ 0×0 - 0×0 ]
Now some counterfactual thinking. One possible scenario after KeWaitForMultipleObjects was called to wait for both objects to become signalled (3rd WAIT_TYPE parameter) the free pool slot was allocated or coalesced with SignalState becoming nonzero by coincidence and other members becoming random values and then the second normal object becomes signalled when another thread sets the notification event…
- Dmitry Vostokov @ DumpAnalysis.org -
a bobo address - an address a system will definitely crash at, given an opportunity
Examples: bad0b0b0, 00000000
- Dmitry Vostokov @ DumpAnalysis.org -
- Dmitry Vostokov @ DumpAnalysis.org -
Birds eye view of software traces makes it easier to see their coarse blocked structure:
where further finer structure is discernible and even nested blocks:
Some blocks of output can be seen when scrolling trace viewer output but if a viewer support zooming it is possible to get an overview and jump directly into a Characteristic Message Block, for example, debug messages of repeated attempts to query a database. If a viewer supports message coloring it also helps. Sometimes this technique is useful to ignore bulk messages and start the analysis around block boundaries.
- Dmitry Vostokov @ TraceAnalysis.org -
Here is a fragment of a condensed view of a CDF (ETW-based) trace imported into MS Word:
- Dmitry Vostokov @ TraceAnalysis.org -
Suppose we want to write a support tool to troubleshoot a frequent software problem. Would we start with GUI? Most tools are command line tools and we therefore choose Tool Façade pattern to decouple GUI from CLI:
By making our GUI optional we postpone its development until later implementation phase when we want to beautify or simplify the tool user interface. The troubleshooting of software problem we want to automate is quite complex, scattered across several knowledge base articles, with many troubleshooting steps and various checks. We therefore treat different steps as TUWs, Troubleshooting Units of Work (aggregating related ones if necessary) and choose Checklist pattern to coordinate them:
Initially we think that just one checklist is suffice but while studying the domain of troubleshooting we find that the problem usually spans client and server computers, where servers sometimes have client roles as well and different troubleshooting steps and checks are needed. In order to simplify the tool we apply Span Differentiator pattern and our architectural UML component diagram becomes more refined:
We also see that different TUW use similar operations like querying registry values or checking whether certain processes are running and factor them out into separate SMs, Supporting Modules:
Deployment methods are important in troubleshooting where it is not always desirable to change configuration on production servers and we plan for optional Self Extractor pattern:
In diagrams we liberally mixed component and static UML diagramming elements. We also need to design a uniform interface that each TUW module exposes to the main tool coordinator module but we leave it for the part 2.
- Dmitry Vostokov @ DumpAnalysis.org -
Support tools can be packaged for delivery as (not counting optional separate readme and EULA files):
The latter common packaging choice is a pattern itself that I call Self Extractor. Typical examples include sysinternals tools like Process Monitor. Here is a simple unorthodox UML diagram for a single platform tool combining elements from static and component diagrams to illustrate component embedding (not sharing):
SM is a supporting module, for example, a driver, to carry out some task or receive notifications.
Some tools even extract different components depending on the processor architecture, for example, x86 and x64. This is illustrated on a more elaborate UML diagram:
The typical example would be a backward compatible x86 Extractor that extracts an x64 driver (Extractee) for an x64 OS kernel that doesn’t load x86 drivers.
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes we have a managed code exception that was enveloping a handled unmanaged code exception, Mixed (Nested) Exception:
0:000> !analyze -v
[...]
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
FAULTING_THREAD: 00000cfc
[...]
EXCEPTION_OBJECT: !pe 1f9af1ac
Exception object: 1f9af1ac
Exception type: System.AccessViolationException
Message: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
InnerException: <none>
StackTrace (generated):
SP IP Function
0012EF3C 28DD9AF9 DllA!Component.getFirstField()+0×11
[…]
0012EFC8 7B194170 System_Windows_Forms_ni!System.Windows.Forms. Control.OnClick(System.EventArgs)+0×70
0012EFE0 7B6F74B4 System_Windows_Forms_ni!System.Windows.Forms. Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)+0×170
0012F06C 7BA29B66 System_Windows_Forms_ni!System.Windows.Forms. Control.WndProc(System.Windows.Forms.Message ByRef)+0×861516
0012F0C4 7B1D1D6A System_Windows_Forms_ni!System.Windows.Forms. ScrollableControl.WndProc(System.Windows.Forms.Message ByRef)+0×2a
0012F0D0 7B1C8640 System_Windows_Forms_ni!System.Windows.Forms. Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)+0×10
0012F0D8 7B1C85C1 System_Windows_Forms_ni!System.Windows.Forms. Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0×31
0012F0EC 7B1C849A System_Windows_Forms_ni!System.Windows.Forms. NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0×5a
[...]
We see that it was the access violation exception and check the thread with TID cfc:
0:000> kL
ChildEBP RetAddr
0012db54 77d70dde ntdll!KiFastSystemCallRet
0012db58 7b1d8e48 user32!NtUserWaitMessage+0xc
0012dbec 7b1d8937 System_Windows_Forms_ni+0x208e48
0012dc44 7b1d8781 System_Windows_Forms_ni+0x208937
0012dc74 7b6edd1f System_Windows_Forms_ni+0x208781
0012dc8c 7b72246b System_Windows_Forms_ni+0x71dd1f
0012dd18 7b722683 System_Windows_Forms_ni+0x75246b
0012dd58 7b6f77f6 System_Windows_Forms_ni+0x752683
0012dd64 7b6fa27c System_Windows_Forms_ni+0x7277f6
0012f148 77d6f8d2 System_Windows_Forms_ni+0x72a27c
0012f174 77d6f794 user32!InternalCallWinProc+0x23
0012f1ec 77d70008 user32!UserCallWinProcCheckWow+0x14b
0012f250 77d70060 user32!DispatchMessageWorker+0x322
0012f260 0a1412fa user32!DispatchMessageW+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012f27c 578439f7 0xa1412fa
0012f2ec 578430c9 WindowsBase_ni+0x939f7
0012f2f8 5784306c WindowsBase_ni+0x930c9
0012f304 55bed46e WindowsBase_ni+0x9306c
0012f310 55bec76f PresentationFramework_ni+0x1cd46e
0012f334 55bd3aa6 PresentationFramework_ni+0x1cc76f
If there was an exception it must be hidden so we inspect the thread raw stack:
0:000> !teb
TEB at 7ffdf000
ExceptionList: 0012e470
StackBase: 00130000
StackLimit: 0011e000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffdf000
EnvironmentPointer: 00000000
ClientId: 00000b6c . 00000cfc
RpcHandle: 00000000
Tls Storage: 7ffdf02c
PEB Address: 7ffd4000
LastErrorValue: 0
LastStatusValue: c0000139
Count Owned Locks: 0
HardErrorMode: 0
0:000> dps 0011e000 00130000
0011e000 00000000
0011e004 00000000
0011e008 00000000
[...]
0012e72c 00130000
0012e730 0011e000
0012e734 00ee350d
0012e738 0012ea3c
0012e73c 77f299f7 ntdll!KiUserExceptionDispatcher+0xf
0012e740 0012e750
0012e744 0012e76c
0012e748 0012e750
0012e74c 0012e76c
0012e750 c0000005
0012e754 00000000
0012e758 00000000
0012e75c 77f17d89 ntdll!RtlLeaveCriticalSection+0×9
0012e760 00000002
0012e764 00000001
0012e768 00000028
0012e76c 0001003f
0012e770 00000000
0012e774 00000000
0012e778 00000000
0012e77c 00000000
[…]
0:000> .cxr 0012e76c
eax=00000020 ebx=09ca1fa0 ecx=781c1b78 edx=00000001 esi=00000020 edi=09ca1ff8
eip=77f17d89 esp=0012ea38 ebp=0012ea3c iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202
ntdll!RtlLeaveCriticalSection+0x9:
77f17d89 834608ff add dword ptr [esi+8],0FFFFFFFFh ds:0023:00000028=????????
0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0012ea3c 7813e5b5 ntdll!RtlLeaveCriticalSection+0x9
0012ea44 2071c9ba msvcr80!_unlock_file+0x35
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ea68 2071c31e DllB!getType+0×286a
0012ee34 206bbfbc DllB!getType+0×3eb
0012ee68 206c8abb DllC+0xbfbc
0012ee98 79e71ca7 DllC!getFirstField+0×3b
0012f148 77d6f8d2 mscorwks!NDirectGenericStubReturnFromCall
0012f1ec 77d70008 user32!InternalCallWinProc+0×23
0012f240 77db51b9 user32!DispatchMessageWorker+0×322
0012f4a4 79e95feb user32!_W32ExceptionHandler+0×18
0012f4fc 79e968b0 mscorwks!MetaSig::HasRetBuffArg+0×5
0012f50c 79e9643e mscorwks!MetaSig::MetaSig+0×3a
0012f610 79e96534 mscorwks!MethodDesc::CallDescr+0xaf
0012f62c 79e96552 mscorwks!MethodDesc::CallTargetWorker+0×1f
0012f644 79eefa45 mscorwks!MethodDescCallSite::CallWithValueTypes+0×1a
0012f7a8 79eef965 mscorwks!ClassLoader::RunMain+0×223
Therefore we identified DllB and DllC components as suspicious. If we check exception chain we see that .NET runtime registered custom exception handlers:
0:000> !exchain
0012e470: mscorwks!COMPlusNestedExceptionHandler+0 (79edd6d7)
0012f13c: mscorwks!FastNExportExceptHandler+0 (7a00a2e7)
0012f1dc: user32!_except_handler4+0 (77db51ba)
0012f240: user32!_except_handler4+0 (77db51ba)
0012f46c: mscorwks!COMPlusFrameHandler+0 (79edc3bc)
0012f4c0: mscorwks!_except_handler4+0 (79f908a2)
0012f798: mscorwks!_except_handler4+0 (79f908a2)
0012fa04: mscorwks!GetManagedNameForTypeInfo+a680 (7a328d90)
0012fed4: mscorwks!GetManagedNameForTypeInfo+82c8 (7a325a3a)
0012ff20: mscorwks!_except_handler4+0 (79f908a2)
0012ff6c: mscorwks!GetManagedNameForTypeInfo+a6e (7a319ee4)
0012ffc4: ntdll!_except_handler4+0 (77ed9834)
Invalid exception stack at ffffffff
We check that GetManagedNameForTypeInfo+a6e (7a319ee4) is an exception handler indeed:
0:000> .asm no_code_bytes
Assembly options: no_code_bytes
0:000> uf 7a319ee4
msvcr80!__CxxFrameHandler:
78158aeb push ebp
78158aec mov ebp,esp
78158aee sub esp,8
78158af1 push ebx
78158af2 push esi
78158af3 push edi
78158af4 cld
78158af5 mov dword ptr [ebp-4],eax
78158af8 xor eax,eax
78158afa push eax
78158afb push eax
78158afc push eax
78158afd push dword ptr [ebp-4]
78158b00 push dword ptr [ebp+14h]
78158b03 push dword ptr [ebp+10h]
78158b06 push dword ptr [ebp+0Ch]
78158b09 push dword ptr [ebp+8]
78158b0c call msvcr80!__InternalCxxFrameHandler (7815897e)
78158b11 add esp,20h
78158b14 mov dword ptr [ebp-8],eax
78158b17 pop edi
78158b18 pop esi
78158b19 pop ebx
78158b1a mov eax,dword ptr [ebp-8]
78158b1d mov esp,ebp
78158b1f pop ebp
78158b20 ret
mscorwks!__CxxFrameHandler3:
79f5f258 jmp dword ptr [mscorwks!_imp____CxxFrameHandler3 (79e711c4)]
mscorwks!GetManagedNameForTypeInfo+0xa6e:
7a319ee4 mov edx,dword ptr [esp+8]
7a319ee8 lea eax,[edx+0Ch]
7a319eeb mov ecx,dword ptr [edx-30h]
7a319eee xor ecx,eax
7a319ef0 call mscorwks!__security_check_cookie (79e72037)
7a319ef5 mov eax,offset mscorwks!_CT??_R0H+0xc14 (7a319f00)
7a319efa jmp mscorwks!__CxxFrameHandler3 (79f5f258)
- Dmitry Vostokov @ DumpAnalysis.org -
“Simplifying computing with a” crash.
Dmitry Vostokov’s excavative exclamation upon hearing a marketing slogan “I have a crush on you”
- Dmitry Vostokov @ DumpAnalysis.org -
First, I introduce the notion of a ruptured computation that is a relaunched, restarted or resumed process from a checkpoint after a rupture in its fabric, for example, crash, hang, spike and leak forcing termination. DNA can be an abbreviation of a Dump of Nasty Application or Double-ruptures of Nasty Application but I got an enlightenment spike in my head recently after recognizing that C, H, S and L are 4 DNA bases: Crash, Hang, Spike and Leak. Do you see a parallel with the real DNA and its 4 bases: A, T, G, C? Now a sample of a ruptured computation DNA strand (imagine, we launch an application and it crashes, …, crashes, …, crashes, …, spikes, …, crashes, …, hangs, …) can be represented as: …CCCSCH… . What about its second strand? I’m in search of it now but in the mean time I introduce the second notion of RNA (Really Nasty Application or Ruptures of Nasty Application) that is just one strand of …CCCSCH… as needed (real RNA is mostly single-stranded).
- Dmitry Vostokov @ DumpAnalysis.org -
When looking at stack traces seasoned memory dump analysis and software maintenance engineers immediately spot the right function to dig into its source code:
STACK_TEXT:
05b3f514 66ed52d3 08d72fee 08da6fc4 05b3f540 msvcrt!wcscpy+0xe
05b3f53c 77c50193 07516fc8 06637fc4 00000006 ModuleA!Add+0xd8
05b3f568 77cb33e1 66ed51fb 05b3f750 00000006 rpcrt4!Invoke+0x30
05b3f968 77cb1968 08d6afe0 0774cf4c 0660af28 rpcrt4!NdrStubCall2+0x299
[...]
Most will start with ModuleA!Add and examine parameters to wcscpy. This is because wcscpy (UNICODE version of strcpy) is considered as a Well-Tested Function. For the purposes of default analysis via !analyse -v command it is possible to configure WinDbg to ignore our own functions and modules as well if we are sure they were well-tested or pass-through. For details please see the old minidump analysis case study.
- Dmitry Vostokov @ DumpAnalysis.org -
Some modules like drivers or runtime DLLs are always present after some action has happened. I call them Effect Components. It is the last thing to assume them to be the “Cause” components” or “Root Cause” or the so so called “culprit” components. Typical example, is dump disk driver symbolic references found in execution residue on the raw stack of a running bugchecking thread:
0: kd> !thread
THREAD fffffa8002bdebb0 Cid 03c4.03f0 Teb: 000007fffffde000 Win32Thread: fffff900c20f9810 RUNNING on processor 0
IRP List:
fffffa8002b986f0: (0006,0118) Flags: 00060000 Mdl: 00000000
Not impersonating
DeviceMap fffff88005346920
Owning Process fffffa80035bec10 Image: Application.exe
Attached Process N/A Image: N/A
Wait Start TickCount 35246 Ticks: 7 (0:00:00:00.109)
Context Switch Count 1595 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.031
Win32 Start Address Application (0x0000000140002708)
Stack Init fffffa600495ddb0 Current fffffa600495d720
Base fffffa600495e000 Limit fffffa6004955000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Child-SP RetAddr : Call Site
fffffa60`0495d558 fffff800`0186e3ee : nt!KeBugCheckEx
fffffa60`0495d560 fffff800`0186d2cb : nt!KiBugCheckDispatch+0×6e
fffffa60`0495d6a0 fffffa60`03d5917a : nt!KiPageFault+0×20b (TrapFrame @ fffffa60`0495d6a0)
[…]
0: kd> dps fffffa6004955000 fffffa600495e000
fffffa60`04955000 00d4d0c8`00d4d0c8
fffffa60`04955008 00d4d0c8`00d4d0c8
fffffa60`04955010 00d4d0c8`00d4d0c8
[…]
fffffa60`0495c7e0 00000000`00000001
fffffa60`0495c7e8 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c7f0 fffffa80`024c05a8
fffffa60`0495c7f8 fffffa60`02869ad4 dump_dumpata!IdeDumpNotification+0×1a4
fffffa60`0495c800 fffffa60`0495cb00
fffffa60`0495c808 fffff800`0182ff34 nt!output_l+0×6c0
fffffa60`0495c810 fffffa60`02860110 crashdmp!StrBeginningDump
fffffa60`0495c818 fffffa60`0495cb00
fffffa60`0495c820 00000000`00000000
fffffa60`0495c828 fffffa60`02869b18 dump_dumpata!IdeDumpNotification+0×1e8
fffffa60`0495c830 00000000`00000000
fffffa60`0495c838 fffffa60`0495c8c0
fffffa60`0495c840 00000000`00000000
fffffa60`0495c848 fffffa60`00000024
fffffa60`0495c850 00000000`ffffffff
fffffa60`0495c858 00000000`00000000
fffffa60`0495c860 00000000`00000000
fffffa60`0495c868 fffffa60`0495cb00
fffffa60`0495c870 fffffa80`00000000
fffffa60`0495c878 00000000`00000000
fffffa60`0495c880 00000000`00000101
fffffa60`0495c888 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c890 fffffa60`0495cb0f
fffffa60`0495c898 fffff800`0182ff34 nt!output_l+0×6c0
fffffa60`0495c8a0 fffffa60`0495cb0f
fffffa60`0495c8a8 fffffa60`0495cb90
fffffa60`0495c8b0 00000000`00000040
fffffa60`0495c8b8 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c8c0 fffffa80`024c0728
fffffa60`0495c8c8 fffffa80`024c0728
fffffa60`0495c8d0 00000001`00000000
fffffa60`0495c8d8 fffffa60`00000026
fffffa60`0495c8e0 00000000`ffffffff
fffffa60`0495c8e8 00000000`00000000
fffffa60`0495c8f0 fffffa80`00000000
fffffa60`0495c8f8 fffffa60`0495cb90
fffffa60`0495c900 00000000`00000000
fffffa60`0495c908 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c910 00000000`00000000
fffffa60`0495c918 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c920 fffff880`05311010
fffffa60`0495c928 00000000`00000002
fffffa60`0495c930 fffffa60`02875094 dump_SATA_Driver!AhciAdapterControl
fffffa60`0495c938 fffffa80`024c6018
fffffa60`0495c940 fffffa80`024c0728
fffffa60`0495c948 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c950 fffffa80`024c0728
fffffa60`0495c958 00000000`00000000
fffffa60`0495c960 fffffa60`0495ca18
fffffa60`0495c968 00000000`00000000
fffffa60`0495c970 fffffa80`024c0728
fffffa60`0495c978 fffffa60`02876427 dump_SATA_Driver!AhciHwInitialize+0×337
fffffa60`0495c980 fffffa80`024c0be6
fffffa60`0495c988 fffffa60`0286a459 dump_dumpata!IdeDumpWaitOnRequest+0×79
fffffa60`0495c990 00000000`00000000
fffffa60`0495c998 00000000`0000023a
fffffa60`0495c9a0 20474e55`534d4153
fffffa60`0495c9a8 204a4831`36314448
fffffa60`0495c9b0 20202020`20202020
fffffa60`0495c9b8 20202020`20202020
fffffa60`0495c9c0 fffffa80`024c05a8
fffffa60`0495c9c8 fffffa60`02869b18 dump_dumpata!IdeDumpNotification+0×1e8
fffffa60`0495c9d0 00000000`00000000
fffffa60`0495c9d8 fffffa60`0495ca60
fffffa60`0495c9e0 00000000`00000001
fffffa60`0495c9e8 fffffa60`02869396 dump_dumpata!IdeDumpMiniportChannelInitialize+0×236
fffffa60`0495c9f0 fffffa80`024c05a8
fffffa60`0495c9f8 fffffa60`02869ad4 dump_dumpata!IdeDumpNotification+0×1a4
fffffa60`0495ca00 00000000`00000000
fffffa60`0495ca08 fffffa60`0495ca90
fffffa60`0495ca10 00000000`00000001
fffffa60`0495ca18 00000001`00000038
fffffa60`0495ca20 00000000`10010000
fffffa60`0495ca28 00000000`00000003
fffffa60`0495ca30 fffffa80`024c05a8
fffffa60`0495ca38 fffffa60`0286a954 dump_dumpata!AtaPortGetPhysicalAddress+0×2c
fffffa60`0495ca40 fffffa80`024c0728
fffffa60`0495ca48 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495ca50 00000000`00000001
fffffa60`0495ca58 0000003f`022a8856
fffffa60`0495ca60 fffffa80`0000000c
fffffa60`0495ca68 fffffa80`024c0728
fffffa60`0495ca70 00000000`00000200
fffffa60`0495ca78 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495ca80 fffffa80`024c0728
fffffa60`0495ca88 ffff6226`4f5f3eb8
fffffa60`0495ca90 00000000`00000010
fffffa60`0495ca98 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495caa0 fffffa80`024c05a8
fffffa60`0495caa8 fffffa60`02875a0d dump_SATA_Driver!AhciHwStartIo+0×69d
fffffa60`0495cab0 fffffa80`024c0728
fffffa60`0495cab8 00000000`00000000
fffffa60`0495cac0 00000000`00000001
fffffa60`0495cac8 fffff800`018f3dfc nt!DisplayCharacter+0×5c
fffffa60`0495cad0 00000000`00000000
fffffa60`0495cad8 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495cae0 00000000`00010000
fffffa60`0495cae8 00000000`00000000
fffffa60`0495caf0 fffffa60`0495cd10
fffffa60`0495caf8 fffffa60`0495cc00
fffffa60`0495cb00 fffffa80`024c01c0
fffffa60`0495cb08 fffffa60`02875c3f dump_SATA_Driver!AhciHwInterrupt+0×2b
fffffa60`0495cb10 fffffa80`024c05a8
fffffa60`0495cb18 00000000`00000000
fffffa60`0495cb20 00000000`00000000
fffffa60`0495cb28 fffff800`01d406c9 hal!KeStallExecutionProcessor+0×25
fffffa60`0495cb30 00000000`00010000
fffffa60`0495cb38 00000000`00000000
fffffa60`0495cb40 fffffa60`0495cd10
fffffa60`0495cb48 fffffa60`0495cc00
fffffa60`0495cb50 00000000`00000000
fffffa60`0495cb58 fffffa60`0286a429 dump_dumpata!IdeDumpWaitOnRequest+0×49
fffffa60`0495cb60 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cb68 00000000`d8bda325
fffffa60`0495cb70 00000000`00000000
fffffa60`0495cb78 00000000`0000033e
fffffa60`0495cb80 00000000`00000000
fffffa60`0495cb88 fffffa60`028694d2 dump_dumpata!IdeDumpWritePending+0xee
fffffa60`0495cb90 fffffa80`024c0000
fffffa60`0495cb98 fffffa80`024c01c0
fffffa60`0495cba0 00000000`00000000
fffffa60`0495cba8 00000000`00000000
fffffa60`0495cbb0 fffffa80`024c01c0
fffffa60`0495cbb8 fffffa80`01e3c740
fffffa60`0495cbc0 00000000`00010000
fffffa60`0495cbc8 00000000`00000000
fffffa60`0495cbd0 00000000`0c01f000
fffffa60`0495cbd8 fffffa60`0285bca9 crashdmp!WritePageSpanToDisk+0×181
fffffa60`0495cbe0 00000000`83d81000
fffffa60`0495cbe8 00000000`00000000
fffffa60`0495cbf0 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cbf8 00000000`00000002
fffffa60`0495cc00 00000000`00000000
fffffa60`0495cc08 00000000`00030000
fffffa60`0495cc10 00000000`00000000
fffffa60`0495cc18 fffffa60`00441000
fffffa60`0495cc20 fffffa60`00441000
fffffa60`0495cc28 00000000`00010000
fffffa60`0495cc30 00000000`0000c080
fffffa60`0495cc38 00000000`0000c081
fffffa60`0495cc40 00000000`0000c082
fffffa60`0495cc48 00000000`0000c083
fffffa60`0495cc50 00000000`0000c084
fffffa60`0495cc58 00000000`0000c085
fffffa60`0495cc60 00000000`0000c086
fffffa60`0495cc68 00000000`0000c087
fffffa60`0495cc70 00000000`0000c088
fffffa60`0495cc78 00000000`0000c089
fffffa60`0495cc80 00000000`0000c08a
fffffa60`0495cc88 00000000`0000c08b
fffffa60`0495cc90 00000000`0000c08c
fffffa60`0495cc98 00000000`0000c08d
fffffa60`0495cca0 00000000`0000c08e
fffffa60`0495cca8 00000000`0000c08f
fffffa60`0495ccb0 00000000`00000000
fffffa60`0495ccb8 00000000`00000000
fffffa60`0495ccc0 00000000`00000000
fffffa60`0495ccc8 00000000`00000010
fffffa60`0495ccd0 00000000`0000c01d
fffffa60`0495ccd8 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cce0 00000000`0000bf80
fffffa60`0495cce8 00000000`00000001
fffffa60`0495ccf0 00000000`00000000
fffffa60`0495ccf8 fffffa80`01e353d0
fffffa60`0495cd00 fffffa80`01e353f8
fffffa60`0495cd08 fffffa60`0285bacc crashdmp!WriteFullDump+0×70
fffffa60`0495cd10 00000002`3a3d8000
fffffa60`0495cd18 00000000`0000c080
fffffa60`0495cd20 fffffa80`00000000
fffffa60`0495cd28 fffffa60`0285c9c0 crashdmp!CrashdmpWriteRoutine
fffffa60`0495cd30 fffff880`05311010
fffffa60`0495cd38 00000000`00000002
fffffa60`0495cd40 fffffa60`0495cf70
fffffa60`0495cd48 00000000`00000000
fffffa60`0495cd50 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cd58 fffffa60`0285b835 crashdmp!DumpWrite+0xc5
fffffa60`0495cd60 00000000`00000000
fffffa60`0495cd68 00000000`0000000f
fffffa60`0495cd70 00000000`00000001
fffffa60`0495cd78 fffffa60`00000001
fffffa60`0495cd80 fffffa80`02bdebb0
fffffa60`0495cd88 fffffa60`0285b153 crashdmp!CrashdmpWrite+0×57
fffffa60`0495cd90 00000000`00000000
fffffa60`0495cd98 fffffa60`028602f0 crashdmp!StrInitPortDriver
fffffa60`0495cda0 00000000`00000000
fffffa60`0495cda8 fffffa60`02860a00 crashdmp!ContextCopy
fffffa60`0495cdb0 00000000`00000000
fffffa60`0495cdb8 fffff800`01902764 nt!IoWriteCrashDump+0×3f4
fffffa60`0495cdc0 fffffa60`0495ce00
fffffa60`0495cdc8 00000028`00000025
fffffa60`0495cdd0 fffff800`018afd40 nt! ?? ::FNODOBFM::`string’
fffffa60`0495cdd8 00000000`000000d1
fffffa60`0495cde0 fffff880`05311010
fffffa60`0495cde8 00000000`00000002
fffffa60`0495cdf0 00000000`00000000
fffffa60`0495cdf8 fffffa60`03d5917a
fffffa60`0495ce00 202a2a2a`0a0d0a0d
fffffa60`0495ce08 7830203a`504f5453
fffffa60`0495ce10 31443030`30303030
fffffa60`0495ce18 46464646`78302820
fffffa60`0495ce20 31333530`30383846
fffffa60`0495ce28 fffff800`018f5f83 nt!VidDisplayString+0×143
fffffa60`0495ce30 30303030`30300030
fffffa60`0495ce38 2c323030`30303030
fffffa60`0495ce40 30303030`30307830
fffffa60`0495ce48 30303030`30303030
fffffa60`0495ce50 46464678`302c3030
fffffa60`0495ce58 fffff800`018fe040 nt!KiInvokeBugCheckEntryCallbacks+0×80
fffffa60`0495ce60 fffffa80`02bdebb0
fffffa60`0495ce68 fffff800`01921d52 nt!InbvDisplayString+0×72
fffffa60`0495ce70 fffff880`05311000
fffffa60`0495ce78 fffff800`01d406c9 hal!KeStallExecutionProcessor+0×25
fffffa60`0495ce80 00000000`00000001
fffffa60`0495ce88 00000000`0000000a
fffffa60`0495ce90 fffffa60`03d5917a
fffffa60`0495ce98 00000000`40000082
fffffa60`0495cea0 00000000`00000001
fffffa60`0495cea8 fffff800`01922c3e nt!KeBugCheck2+0×92e
fffffa60`0495ceb0 fffff800`000000d1
fffffa60`0495ceb8 00000000`000004d0
fffffa60`0495cec0 fffff800`01a43640 nt!KiProcessorBlock
fffffa60`0495cec8 00000000`0000000a
fffffa60`0495ced0 fffffa60`03d5917a
fffffa60`0495ced8 fffffa60`0495cf70
fffffa60`0495cee0 fffffa80`02bdebb0
fffffa60`0495cee8 00000000`00000000
fffffa60`0495cef0 00000000`00000000
fffffa60`0495cef8 fffffa80`02bdebb0
fffffa60`0495cf00 00000000`c21a6d00
fffffa60`0495cf08 00000000`00000000
fffffa60`0495cf10 fffff800`0198e7a0 nt!KiInitialPCR+0×2a0
fffffa60`0495cf18 fffff800`0198e680 nt!KiInitialPCR+0×180
fffffa60`0495cf20 fffffa80`02bb7320
fffffa60`0495cf28 00000000`00000000
fffffa60`0495cf30 00000000`00000000
fffffa60`0495cf38 fffff960`00000003
fffffa60`0495cf40 fffffa60`0495e000
fffffa60`0495cf48 fffffa60`04955000
fffffa60`0495cf50 00000001`c0643000
fffffa60`0495cf58 00000000`00000000
fffffa60`0495cf60 fffff900`c06ca53c
fffffa60`0495cf68 fffffa60`0495d090
fffffa60`0495cf70 00000000`00000000
fffffa60`0495cf78 00000000`00000000
fffffa60`0495cf80 00000000`00000000
fffffa60`0495cf88 00000000`00000000
fffffa60`0495cf90 00000000`00000000
fffffa60`0495cf98 00000000`00000000
fffffa60`0495cfa0 00001f80`0010000f
fffffa60`0495cfa8 0053002b`002b0010
fffffa60`0495cfb0 00000286`0018002b
fffffa60`0495cfb8 00000000`00000000
fffffa60`0495cfc0 00000000`00000000
fffffa60`0495cfc8 00000000`00000000
fffffa60`0495cfd0 00000000`00000000
fffffa60`0495cfd8 00000000`00000000
fffffa60`0495cfe0 00000000`00000000
fffffa60`0495cfe8 fffffa60`0495d660
fffffa60`0495cff0 00000000`0000000a
fffffa60`0495cff8 fffff880`05311010
fffffa60`0495d000 fffff880`05311010
fffffa60`0495d008 fffffa60`0495d558
fffffa60`0495d010 fffffa60`0495d720
fffffa60`0495d018 fffffa80`02b986f0
fffffa60`0495d020 fffffa80`02b98720
fffffa60`0495d028 00000000`00000002
fffffa60`0495d030 00000000`00000000
fffffa60`0495d038 fffffa60`03d5917a
fffffa60`0495d040 00000000`000001f1
fffffa60`0495d048 fffffa80`026a9df0
fffffa60`0495d050 00000000`00000001
fffffa60`0495d058 00000000`83360018
fffffa60`0495d060 fffffa80`02b3ee40
fffffa60`0495d068 fffff800`0186e650 nt!KeBugCheckEx
fffffa60`0495d070 00000000`00000000
fffffa60`0495d078 00000000`00000000
fffffa60`0495d080 00000000`00000000
fffffa60`0495d088 00000000`00000000
fffffa60`0495d090 00000000`00000000
fffffa60`0495d098 00000000`00000000
fffffa60`0495d0a0 00000000`00000000
[…]
If a BSOD was reported after installing new drivers we shouldn’t suspect SATA_Driver package here because its components would almost always be present on any bugcheck thread as referenced after a bugcheck cause. There presence is the “effect”. This example might seem trivial and pointless but I’ve seen some memory dump analysis conclusions based on the reversal of causes and effects.
- Dmitry Vostokov @ DumpAnalysis.org -
What prompted me to found this discipline (that is supposed to be a sister discipline of software criminology, software security, secure code construction and software defect construction) is understanding that some software components are innocent victims of other component coding mistakes or deliberate subversion and some start as a part of crimeware but eventually become victims themselves (they crash, hang, spike, leak, are dumped, subverted, etc.). I would also like to borrow and reuse the neglected term victimware here in a broad sense. More on this later as I have to switch to software trace analysis patterns.
- Dmitry Vostokov @ DumpAnalysis.org -
One ETW trace pointed to a set of intermittent symptoms (messages were simplified for this post):
# PID TID Message
[...]
31278 2300 7060 RequestXMLData entry
31281 2300 7060 RequestXMLData: XML error
[...]
Searching for issues having this error only pointed to a case with a mixed software product environment where some servers had the product version X and other servers the product version X+1. However, in the new case the customer claimed that he had only the product version X+1 on all production servers. We insisted and, after the closer inspection, servers with the product X were found…
- Dmitry Vostokov @ TraceAnalysis.org -
The domain of some troubleshooting and debugging tools sometimes spans several computers, for example, a client, a server, and a middleware server (or a frontware server like a web server). Because such tools are small, there is no need to devise yet another distributed architecture but force the tool to differentiate its functionality based on a command line parameters. For example, the following tool implements Checklist pattern and chooses the right checklist from a checklist repository based on a command line option:
c:\> GreatTool -client
c:\> GreatTool -server
This nicely fits with a case when a server is also a client to another server and we need to troubleshoot its client and server functionality on the same physical computer.
- Dmitry Vostokov @ DumpAnalysis.org -
Many troubleshooting tools and their components (like TUW modules from checklists) reuse the same code over and over, for example, querying registry keys or environment variables. It is beneficial to have a package of supporting modules (or a library) for code reuse purposes, hence the name of this pattern, Supporting Module (SM). In the context of Checklist pattern we can refine its UML component diagram to highlight that TUWs are using some shared SM components:
- Dmitry Vostokov @ DumpAnalysis.org -
Computation “is a brief gasp between one” crash dump “and another.”
Ezra Weston Loomis Pound, Subbugtated from Poetry: The Basics (Chapter 3), Jeffrey Wainwright
- Dmitry Vostokov @ DumpAnalysis.org -
Memory Analysis and Debugging Institute (MA&DI), DA+TA Portal (DumpAnalysis.org + TraceAnalysis.org) and OpenTask establish
Software Maintenance Institute (SMI)
More details about R&D Institute of Software Maintenance will be announced soon.
- Dmitry Vostokov @ DumpAnalysis.org -
